search

nicksherron / proxi

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Apache License 2.0
32 stars 4 forks source link

checks if the proxy bypass a website protected by CloudFlare #1

Open ghost opened 4 years ago

ghost commented 4 years ago

Hi,

Hope you are all well !

Is it possible to add an option to proxi for checking if a proxy bypass a website protected by CloudFlare ?

For doing so, we need to find a cloudflare website (list from https://www.wappalyzer.com/technologies/cdn/cloudflare) and http://getbootstrap.com/ sounds like a good choice ?!

And, from my understanding we can add it to the PROXI_JUDGES env variables but can we add it in the database as a specific field/property in the database ?

Cheers, X

nicksherron commented 4 years ago

Hey @x0rzkov ,

First off, thanks for your feedback and interest in the project and I apologize for my slow response.

My mother has become terminally ill and its been a struggle trying to juggle paid projects let alone anything open-source, so my attention I can give to this is unfortunately limited at the moment. However, I do plan on continuing development once things get settled and will try to keep up with any issues and pull requests in the meantime.

And finally to answer your question!

This is actually a really good idea and something I have thought about as well. This currently wouldn't be possible because the proxy check process is hardcoded to expect the response from a PROXI_JUDGES request to be a httpbin api like json response, and needs to be rewritten/designed (really the whole project needs this). You can checkout the code for it at internal/check.go#L279 .

The original plan was to let allow any url (or multiple different urls) to be used for checks. What do you about having the option of supplying a list of urls with labels as a config file that the checks run on store the results under each corresponding label? That way if you were gathering proxies for scraping a specific site, for example, you could test them out beforehand.