Disable mitmproxy until we can come up with a better solution

openaustralia / morph

Take the hassle out of web scraping

https://morph.io

GNU Affero General Public License v3.0

461 stars 74 forks source link

Disable mitmproxy until we can come up with a better solution #1219

Closed mlandauer closed 5 years ago

mlandauer commented 5 years ago

mitmproxy has causes lots of issues and provides relatively little value to users.

I think, despite much intermittent effort, we've never managed to fix all the subtle problems it introduces including some performance issues.

So, let's disable it and then figure out what to do next (if anything)

mlandauer commented 5 years ago

Added an option for individual scrapers to bypass the proxy. This way I can do some testing on the problematic scrapers and make sure everything does what is expected. So far so good. For example, the scraper https://morph.io/planningalerts-scrapers/shellharbour which was getting 502 (bad gateway) errors, now works when the proxy is disabled.