openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

Return network use of scraper by using a proxy #16

Open mlandauer opened 5 years ago

mlandauer commented 5 years ago

But don't make the proxy a transparent one.

From morph we learned that it's not really possible to make the proxy truly transparent to the user because some languages just don't make it easy to update certificates.

Instead it makes sense to automatically set http_proxy and https_proxy environment variables and tell the user that they need to use the proxy to access the outside world. Most of the time they won't need to do anything but in case something goes wrong they have a way to understand what is happening and debug it.

mlandauer commented 5 years ago

Related to #15