search

claffin / cloudproxy

Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.
https://cloudproxy.io/
MIT License
1.4k stars 79 forks source link

Multiple regions & Historical reporting #61

Open mrahmadt opened 2 years ago

mrahmadt commented 2 years ago

Hello

Thank you very much for this great script, simple and can be a replacement for scraproxy

Is it possible to define multiple regions? for example, I want to have 3-5 regions with digitalocean and cloudproxy will randomly create VMs on them

my second question is there any log file or report that I can use to check how many VMs has been created, the duration, and the period? so I can compare my hosting cost let's say on weekly bases and decide which cloud provider is better for me?

claffin commented 2 years ago

Hey,

Thanks for the comment.

Multiple regions make sense. I'll look to implement in the future.

Log files/report are more complicated as this application is stateless. That data would need to be stored somewhere. I'm not sure on the best solution, a s3 bucket or a database maybe. I agree it would be useful to keep metrics and logs, but needs to be stored somewhere.

ajtatum commented 1 year ago

Hi there, I stumbled upon this project and it looks great! Awesome work! I was just curious if this is still on the radar or if the project is considered inactive?