Develop as Firefox Add-on to Track All HTTP Requests

FrancescoSTL / Site-Sonar

A browser extension which silently crowd-sources ad performance as you browse. Let's put an end to bad ads.

http://site-sonar.com

Mozilla Public License 2.0

9 stars 2 forks source link

Develop as Firefox Add-on to Track All HTTP Requests #1

Closed FrancescoSTL closed 8 years ago

FrancescoSTL commented 8 years ago

The current method for collecting potential HTTP requests and testing their speed seems flawed, as we are not loading the javascript on-page when we are making the requests, and it is likely that we are missing requests because of it. The original reasoning behind the current method was that browser automation is expensive, especially when crawling hundreds (if not thousands) of pages.

FrancescoSTL commented 8 years ago

After investigating a bit, it seems the best implementation will be along the following lines:

Add-on sdk extension for Firefox in order to gather all HTTP requests on any given page.
Tabs API to navigate between pages
Disconnect's blacklist for determining which resources to benchmark

FrancescoSTL commented 8 years ago

Update; add-on is now being developed as a web-extension because all functionality is supported (also so things don't start breaking).

General flow for the crawler:

Trigger page load
When page is requested, start listeners and log results
When page is completed via tabs onupdated, remove sendheaders listener, stop logging results, and set timeout for checking for uncompleted requests
When timeout completes, remove onheadersrecieved listener, iterate through the logged results, and log them as errors/timeouts
Trigger next page load

FrancescoSTL commented 8 years ago

Merged in #3