Generic MITM proxy server

[chfoo]

Things such as the cookie jar handling and request/response event logging were taken out of the proxy and left for handling in other classes. The intended use case was that request/response handling were to be processed by a PhantomJS processing class. This turned to be a bad thing since PhantomJS doesn't emit proper request/response events, is very outdated, and useful interprocess communication is virtually nonexistent.

Instead, the proxy should handle request/response events, rejecting URLs by forceful closing of connections, cookie jar handling, changing the user-agent, etc. This will allow the user to point anything at Wpull's proxy and make it easier to support youtube-dl and other junk.

Edit:

• [x] call scripting hooks so they are aware of it
• [x] allow scripting hooks to change the behavior (abort connection, etc)
• [x] insert URLs fetched into the database so Wpull knows it doesn't need to download them again if the URLs are discovered during scraping
• [ ] add method for user to start up a standalone server

[ikreymer]

+1 I was realized wpull has MITM https proxy support, and thought it could run in standalone mode, and then point a browser or curl through it, but looks it only works with phantomjs yet, right? I tried python -m wpull.proxy.server which seems to work but I assume is just for testing. Would be great to have this!

[chfoo]

@ikreymer There's options provided for it, but it's still a work in progress. There's no way to tell Wpull to start up and just wait for connections yet.