philippta / flyscrape

Flyscrape is a command-line web scraping tool designed for those without advanced programming skills.
https://flyscrape.com
Mozilla Public License 2.0
1.02k stars 29 forks source link

Blacklist non-working Proxies #36

Open philippta opened 7 months ago

philippta commented 7 months ago

A feature should be added to the proxy module that blacklists non-functioning proxies. A proxy is considered non-functioning if the network connection can't be established or if the HTTP status code 407 Proxy Authentication Require is present.

Non-functioning proxies should be stored in a file called <scriptname>.badproxy in the format of one proxy URL per line. This follows the naming convention of the cache file.