search

fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.
http://scrapoxy.io
MIT License
2.01k stars 235 forks source link

aws instances being created and destroyed on its own over and over #132

Closed shairyar closed 10 months ago

shairyar commented 6 years ago

Hello,

I am using the latest version of Scrapoxy and I have proxy authentication in place. The moment I start sending requests via scrapoxy it shuts down the alive instance (after sending the first request successfully) and creates a new instance, shuts that down, creates a new one and goes on forever.

I cant seem to figure out why that is, I thought I was getting banned so I did an experiment by sending the request to another domain I own to see what happens, but the behaviour is same, the first request is a successful one and right after that it shuts down that instance and start creating and destroying aws instances for ever. This continues even if i am no longer sending requests via scrapoxy.

I have to shut down the scrapoxy so it stops creating the destroying the instances.

Here is what I am seeing in log but there is nothing in it that can tell me whats happening (as far as my knowledge goes).

0|scrapoxy  | 2018-09-06T12:33:43.194Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:33:44.869Z - debug: [Manager] checkInstances: add: i-07159ff2fb8f914e5@52.62.130.92:3128
0|scrapoxy  | 2018-09-06T12:33:44.872Z - debug: [Instance/i-07159ff2fb8f914e5] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:33:44.873Z - debug: [Manager] checkInstances: add: i-0c7588241c0a21983@13.211.202.120:3128
0|scrapoxy  | 2018-09-06T12:33:44.873Z - debug: [Instance/i-0c7588241c0a21983] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:33:44.874Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:33:53.195Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:33:54.721Z - debug: [Manager] checkInstances: remove: i-0d6ab38443872e690@13.211.136.71:3128
0|scrapoxy  | 2018-09-06T12:33:54.723Z - debug: [Instance/i-0d6ab38443872e690] changeAlive: true => false
0|scrapoxy  | 2018-09-06T12:33:54.724Z - debug: [Instance/i-0d6ab38443872e690] Instance is down. Crash timer starts
0|scrapoxy  | 2018-09-06T12:33:54.725Z - debug: [Manager] checkInstances: remove: i-07159ff2fb8f914e5@52.62.130.92:3128
0|scrapoxy  | 2018-09-06T12:33:54.726Z - debug: [Instance/i-07159ff2fb8f914e5] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:33:54.727Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:33:54.728Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:33:54.729Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:34:03.197Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:04.907Z - debug: [Manager] checkInstances: add: i-0821f7b91cc93bfdb@13.238.185.237:3128
0|scrapoxy  | 2018-09-06T12:34:04.909Z - debug: [Instance/i-0821f7b91cc93bfdb] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:04.910Z - debug: [Manager] checkInstances: add: i-0f60bbdaa1abe2290@13.236.0.183:3128
0|scrapoxy  | 2018-09-06T12:34:04.911Z - debug: [Instance/i-0f60bbdaa1abe2290] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:04.912Z - debug: [Instance/i-0c7588241c0a21983] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:34:04.913Z - debug: [Instance/i-0c7588241c0a21983] autorestart in 35493934 secs
0|scrapoxy  | 2018-09-06T12:34:04.914Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:34:13.199Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:14.796Z - debug: [Instance/i-0821f7b91cc93bfdb] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:34:14.798Z - debug: [Instance/i-0821f7b91cc93bfdb] autorestart in 16211008 secs
0|scrapoxy  | 2018-09-06T12:34:14.799Z - debug: [Manager] checkInstances: remove: i-0c7588241c0a21983@13.211.202.120:3128
0|scrapoxy  | 2018-09-06T12:34:14.804Z - debug: [Instance/i-0c7588241c0a21983] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:14.806Z - debug: [Manager] checkInstances: remove: i-0f60bbdaa1abe2290@13.236.0.183:3128
0|scrapoxy  | 2018-09-06T12:34:14.807Z - debug: [Instance/i-0f60bbdaa1abe2290] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:14.808Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:34:14.809Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:34:14.809Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:34:23.202Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:24.942Z - debug: [Manager] checkInstances: add: i-05aa9e26e9dd4af37@13.236.188.228:3128
0|scrapoxy  | 2018-09-06T12:34:24.945Z - debug: [Instance/i-05aa9e26e9dd4af37] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:24.952Z - debug: [Manager] checkInstances: add: i-025ffdb8c3f355fcc@54.252.250.152:3128
0|scrapoxy  | 2018-09-06T12:34:24.953Z - debug: [Instance/i-025ffdb8c3f355fcc] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:24.954Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:34:33.205Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:34.794Z - debug: [Manager] checkInstances: remove: i-05aa9e26e9dd4af37@13.236.188.228:3128
0|scrapoxy  | 2018-09-06T12:34:34.795Z - debug: [Instance/i-05aa9e26e9dd4af37] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:34.796Z - debug: [Manager] checkInstances: remove: i-025ffdb8c3f355fcc@54.252.250.152:3128
0|scrapoxy  | 2018-09-06T12:34:34.797Z - debug: [Instance/i-025ffdb8c3f355fcc] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:34.797Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:34:34.798Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:34:34.798Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:34:34.802Z - debug: [Instance/i-0821f7b91cc93bfdb] checkAlive: false / -
0|scrapoxy  | 2018-09-06T12:34:34.802Z - debug: [Pinger] ping: hostname=13.238.185.237 / port=3128
0|scrapoxy  | 2018-09-06T12:34:35.007Z - debug: [Instance/i-0821f7b91cc93bfdb] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:43.207Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:44.792Z - debug: [Manager] checkInstances: add: i-0d0191e160a96b6a7
0|scrapoxy  | 2018-09-06T12:34:44.794Z - debug: [Instance/i-0d0191e160a96b6a7] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:44.795Z - debug: [Manager] checkInstances: add: i-0f505543b16b83535
0|scrapoxy  | 2018-09-06T12:34:44.796Z - debug: [Instance/i-0f505543b16b83535] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:44.797Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:34:53.209Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:54.844Z - debug: [Manager] checkInstances: remove: i-0821f7b91cc93bfdb@13.238.185.237:3128
0|scrapoxy  | 2018-09-06T12:34:54.845Z - debug: [Instance/i-0821f7b91cc93bfdb] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:54.846Z - debug: [Manager] checkInstances: remove: i-0d0191e160a96b6a7
0|scrapoxy  | 2018-09-06T12:34:54.851Z - debug: [Instance/i-0d0191e160a96b6a7] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:54.852Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:34:54.852Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:34:54.852Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:35:03.210Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:35:04.883Z - debug: [Instance/i-0f505543b16b83535] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:35:04.884Z - debug: [Instance/i-0f505543b16b83535] autorestart in 19699566 secs
0|scrapoxy  | 2018-09-06T12:35:04.885Z - debug: [Manager] checkInstances: add: i-07219e8a98995e212@54.206.123.65:3128
0|scrapoxy  | 2018-09-06T12:35:04.886Z - debug: [Instance/i-07219e8a98995e212] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:04.887Z - debug: [Manager] checkInstances: add: i-05c592796ecdf86aa@13.236.162.46:3128
0|scrapoxy  | 2018-09-06T12:35:04.887Z - debug: [Instance/i-05c592796ecdf86aa] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:04.889Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:35:13.213Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:35:14.829Z - debug: [Instance/i-07219e8a98995e212] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:35:14.831Z - debug: [Instance/i-07219e8a98995e212] autorestart in 33868750 secs
0|scrapoxy  | 2018-09-06T12:35:14.832Z - debug: [Manager] checkInstances: remove: i-0f505543b16b83535@13.211.178.121:3128
0|scrapoxy  | 2018-09-06T12:35:14.833Z - debug: [Instance/i-0f505543b16b83535] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:14.834Z - debug: [Manager] checkInstances: remove: i-05c592796ecdf86aa@13.236.162.46:3128
0|scrapoxy  | 2018-09-06T12:35:14.835Z - debug: [Instance/i-05c592796ecdf86aa] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:14.836Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:35:14.836Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:35:14.837Z - debug: [ProviderAWSEC2] createInstances: count=2

I will really appreciate a quick response on this one as I am not able to use scrapoxy at all.

MaZZly commented 5 years ago

I have had something similar happen that after a certain time it will start killing and starting the nodes, also the scrapoxy script itself seems to be dying a bunch. What usually helps is to kill the server and start it up again. Not sure why it does that tho.

fabienvauchelles commented 1 year ago

connectivity issue. solved in 4.0.0

fabienvauchelles commented 10 months ago

Hey there! 🌟 Exciting news! Scrapoxy 4 is ready to rock 🚀. Check it out at Scrapoxy.io (explore the "get started" guide, deployment documentation, and more 📚). I can't wait to hear your feedback on this new version! Send me your coolest screenshots with as many proxies as possible! 📸💻 Join the Discord community if you have any questions or just want to chat. You can also open a GitHub issue for any bug or feature request 🐞✨. See you soon! 😎 Fabien