fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers πŸ•ΈοΈ. It also smartly handles traffic routing πŸ”€ to minimize bans and increase success rates πŸš€.
http://scrapoxy.io
MIT License
2.01k stars 235 forks source link

aws instances being created and destroyed on its own over and over #132

Closed shairyar closed 10 months ago

shairyar commented 6 years ago

Hello,

I am using the latest version of Scrapoxy and I have proxy authentication in place. The moment I start sending requests via scrapoxy it shuts down the alive instance (after sending the first request successfully) and creates a new instance, shuts that down, creates a new one and goes on forever.

I cant seem to figure out why that is, I thought I was getting banned so I did an experiment by sending the request to another domain I own to see what happens, but the behaviour is same, the first request is a successful one and right after that it shuts down that instance and start creating and destroying aws instances for ever. This continues even if i am no longer sending requests via scrapoxy.

I have to shut down the scrapoxy so it stops creating the destroying the instances.

Here is what I am seeing in log but there is nothing in it that can tell me whats happening (as far as my knowledge goes).

0|scrapoxy  | 2018-09-06T12:33:43.194Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:33:44.869Z - debug: [Manager] checkInstances: add: i-07159ff2fb8f914e5@52.62.130.92:3128
0|scrapoxy  | 2018-09-06T12:33:44.872Z - debug: [Instance/i-07159ff2fb8f914e5] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:33:44.873Z - debug: [Manager] checkInstances: add: i-0c7588241c0a21983@13.211.202.120:3128
0|scrapoxy  | 2018-09-06T12:33:44.873Z - debug: [Instance/i-0c7588241c0a21983] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:33:44.874Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:33:53.195Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:33:54.721Z - debug: [Manager] checkInstances: remove: i-0d6ab38443872e690@13.211.136.71:3128
0|scrapoxy  | 2018-09-06T12:33:54.723Z - debug: [Instance/i-0d6ab38443872e690] changeAlive: true => false
0|scrapoxy  | 2018-09-06T12:33:54.724Z - debug: [Instance/i-0d6ab38443872e690] Instance is down. Crash timer starts
0|scrapoxy  | 2018-09-06T12:33:54.725Z - debug: [Manager] checkInstances: remove: i-07159ff2fb8f914e5@52.62.130.92:3128
0|scrapoxy  | 2018-09-06T12:33:54.726Z - debug: [Instance/i-07159ff2fb8f914e5] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:33:54.727Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:33:54.728Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:33:54.729Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:34:03.197Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:04.907Z - debug: [Manager] checkInstances: add: i-0821f7b91cc93bfdb@13.238.185.237:3128
0|scrapoxy  | 2018-09-06T12:34:04.909Z - debug: [Instance/i-0821f7b91cc93bfdb] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:04.910Z - debug: [Manager] checkInstances: add: i-0f60bbdaa1abe2290@13.236.0.183:3128
0|scrapoxy  | 2018-09-06T12:34:04.911Z - debug: [Instance/i-0f60bbdaa1abe2290] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:04.912Z - debug: [Instance/i-0c7588241c0a21983] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:34:04.913Z - debug: [Instance/i-0c7588241c0a21983] autorestart in 35493934 secs
0|scrapoxy  | 2018-09-06T12:34:04.914Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:34:13.199Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:14.796Z - debug: [Instance/i-0821f7b91cc93bfdb] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:34:14.798Z - debug: [Instance/i-0821f7b91cc93bfdb] autorestart in 16211008 secs
0|scrapoxy  | 2018-09-06T12:34:14.799Z - debug: [Manager] checkInstances: remove: i-0c7588241c0a21983@13.211.202.120:3128
0|scrapoxy  | 2018-09-06T12:34:14.804Z - debug: [Instance/i-0c7588241c0a21983] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:14.806Z - debug: [Manager] checkInstances: remove: i-0f60bbdaa1abe2290@13.236.0.183:3128
0|scrapoxy  | 2018-09-06T12:34:14.807Z - debug: [Instance/i-0f60bbdaa1abe2290] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:14.808Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:34:14.809Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:34:14.809Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:34:23.202Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:24.942Z - debug: [Manager] checkInstances: add: i-05aa9e26e9dd4af37@13.236.188.228:3128
0|scrapoxy  | 2018-09-06T12:34:24.945Z - debug: [Instance/i-05aa9e26e9dd4af37] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:24.952Z - debug: [Manager] checkInstances: add: i-025ffdb8c3f355fcc@54.252.250.152:3128
0|scrapoxy  | 2018-09-06T12:34:24.953Z - debug: [Instance/i-025ffdb8c3f355fcc] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:24.954Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:34:33.205Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:34.794Z - debug: [Manager] checkInstances: remove: i-05aa9e26e9dd4af37@13.236.188.228:3128
0|scrapoxy  | 2018-09-06T12:34:34.795Z - debug: [Instance/i-05aa9e26e9dd4af37] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:34.796Z - debug: [Manager] checkInstances: remove: i-025ffdb8c3f355fcc@54.252.250.152:3128
0|scrapoxy  | 2018-09-06T12:34:34.797Z - debug: [Instance/i-025ffdb8c3f355fcc] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:34.797Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:34:34.798Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:34:34.798Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:34:34.802Z - debug: [Instance/i-0821f7b91cc93bfdb] checkAlive: false / -
0|scrapoxy  | 2018-09-06T12:34:34.802Z - debug: [Pinger] ping: hostname=13.238.185.237 / port=3128
0|scrapoxy  | 2018-09-06T12:34:35.007Z - debug: [Instance/i-0821f7b91cc93bfdb] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:43.207Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:44.792Z - debug: [Manager] checkInstances: add: i-0d0191e160a96b6a7
0|scrapoxy  | 2018-09-06T12:34:44.794Z - debug: [Instance/i-0d0191e160a96b6a7] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:44.795Z - debug: [Manager] checkInstances: add: i-0f505543b16b83535
0|scrapoxy  | 2018-09-06T12:34:44.796Z - debug: [Instance/i-0f505543b16b83535] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:44.797Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:34:53.209Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:34:54.844Z - debug: [Manager] checkInstances: remove: i-0821f7b91cc93bfdb@13.238.185.237:3128
0|scrapoxy  | 2018-09-06T12:34:54.845Z - debug: [Instance/i-0821f7b91cc93bfdb] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:54.846Z - debug: [Manager] checkInstances: remove: i-0d0191e160a96b6a7
0|scrapoxy  | 2018-09-06T12:34:54.851Z - debug: [Instance/i-0d0191e160a96b6a7] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:34:54.852Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:34:54.852Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:34:54.852Z - debug: [ProviderAWSEC2] createInstances: count=2
0|scrapoxy  | 2018-09-06T12:35:03.210Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:35:04.883Z - debug: [Instance/i-0f505543b16b83535] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:35:04.884Z - debug: [Instance/i-0f505543b16b83535] autorestart in 19699566 secs
0|scrapoxy  | 2018-09-06T12:35:04.885Z - debug: [Manager] checkInstances: add: i-07219e8a98995e212@54.206.123.65:3128
0|scrapoxy  | 2018-09-06T12:35:04.886Z - debug: [Instance/i-07219e8a98995e212] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:04.887Z - debug: [Manager] checkInstances: add: i-05c592796ecdf86aa@13.236.162.46:3128
0|scrapoxy  | 2018-09-06T12:35:04.887Z - debug: [Instance/i-05c592796ecdf86aa] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:04.889Z - debug: [Manager] adjustInstances: required:3 / actual:3
0|scrapoxy  | 2018-09-06T12:35:13.213Z - debug: [Manager] checkInstances
0|scrapoxy  | 2018-09-06T12:35:14.829Z - debug: [Instance/i-07219e8a98995e212] checkAlive every 20000 secs
0|scrapoxy  | 2018-09-06T12:35:14.831Z - debug: [Instance/i-07219e8a98995e212] autorestart in 33868750 secs
0|scrapoxy  | 2018-09-06T12:35:14.832Z - debug: [Manager] checkInstances: remove: i-0f505543b16b83535@13.211.178.121:3128
0|scrapoxy  | 2018-09-06T12:35:14.833Z - debug: [Instance/i-0f505543b16b83535] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:14.834Z - debug: [Manager] checkInstances: remove: i-05c592796ecdf86aa@13.236.162.46:3128
0|scrapoxy  | 2018-09-06T12:35:14.835Z - debug: [Instance/i-05c592796ecdf86aa] changeAlive: false => false
0|scrapoxy  | 2018-09-06T12:35:14.836Z - debug: [Manager] adjustInstances: required:3 / actual:1
0|scrapoxy  | 2018-09-06T12:35:14.836Z - debug: [Manager] adjustInstances: add 2 instances
0|scrapoxy  | 2018-09-06T12:35:14.837Z - debug: [ProviderAWSEC2] createInstances: count=2

I will really appreciate a quick response on this one as I am not able to use scrapoxy at all.

MaZZly commented 5 years ago

I have had something similar happen that after a certain time it will start killing and starting the nodes, also the scrapoxy script itself seems to be dying a bunch. What usually helps is to kill the server and start it up again. Not sure why it does that tho.

fabienvauchelles commented 1 year ago

connectivity issue. solved in 4.0.0

fabienvauchelles commented 10 months ago

Hey there! 🌟 Exciting news! Scrapoxy 4 is ready to rock πŸš€. Check it out at Scrapoxy.io (explore the "get started" guide, deployment documentation, and more πŸ“š). I can't wait to hear your feedback on this new version! Send me your coolest screenshots with as many proxies as possible! πŸ“ΈπŸ’» Join the Discord community if you have any questions or just want to chat. You can also open a GitHub issue for any bug or feature request 🐞✨. See you soon! 😎 Fabien