Closed FrancescoSTL closed 8 years ago
After investigating a bit, it seems the best implementation will be along the following lines:
Update; add-on is now being developed as a web-extension because all functionality is supported (also so things don't start breaking).
General flow for the crawler:
Merged in #3
The current method for collecting potential HTTP requests and testing their speed seems flawed, as we are not loading the javascript on-page when we are making the requests, and it is likely that we are missing requests because of it. The original reasoning behind the current method was that browser automation is expensive, especially when crawling hundreds (if not thousands) of pages.