Closed hwamil closed 3 years ago
Thanks for putting this together, I will try and review later today, but any reason why it says 57 files changed? It doesn't look like there are any changes to some files.
My vm shit on me so I just copied the folder over to another instance and I think that might be the culprit. I should probably do another fresh branch and make a new pull request :(
I took @DakkJaniels's advice and created an ItemHandler class that holds a single queue of all the items that gets assigned to each session/proxy.
It instantiates n sessions for n of proxies, and ItemHandler.items is a itertools.cycle list that is a circular array. Each time a session finishes a loop it uses the built in next() method to grab the "next" item in the queue. This way it is possible to rapidly check the same items if you have enough proxies.