Random Proxy Rotation

[hwamil]

Tried adding random rotation to the proxies.

Changed the format of proxies.json to just a list to simplify for myself.

In AmazonMonitoringHandler's init, each proxy is placed in a nested dictionary with the idx number as the parent key and the inner dict holding proxy as key and time.time() as value. eg) {31: {user:pw@ip:port : time.time()}}

The function AmazonMonitor.get_new_proxy() is ran in each stock_check function where it uses randint to generate a number between 0 and (number of proxies - 1), which is used to call the dictionary holding the respective proxy address and the last time it was used. All of this is in a while loop that breaks when the condition that the last access time of the time value is greater than 6 seconds.

It's mega clunky and I still don't have a very clear idea how async works and would love some advices to improve or change the design of this. Thanks.

[hwamil]

After testing it's a little faster than the current alpha branch but may need tests with lower response latencies to know for sure.

[DakkJaniels]

This works?

[hwamil]

It appears so to me at least through the cli with the log statements, but not hundo what's going on under the hood. I had a lot of trouble figuring out how to change the connector (had to use @property and _connector) each loop so I hope it's working as intended. It's not too many lines so if you could skim it over and critique it I'd appreciate it.

[bravochar]

I've looked at something like this experimentally before. I think that switching the proxy of a session is more suspicious and more tenuous than setting up a session for each proxy and pseudo-randomly choosing a session (making sure that we're hitting all proxies evenly) for each stock check. If we're using session cookies in each session, then swapping proxies might eventually raise flags. It also makes the code cleaner to be choosing a random session instead of reassigning proxies.

All of that was back with requests, so I'd need to take a closer look to see if it's feasible with asyncio.

[DakkJaniels]

I've looked at something like this experimentally before. I think that switching the proxy of a session is more suspicious and more tenuous than setting up a session for each proxy and pseudo-randomly choosing a session (making sure that we're hitting all proxies evenly) for each stock check. If we're using session cookies in each session, then swapping proxies might eventually raise flags. It also makes the code cleaner to be choosing a random session instead of reassigning proxies.

This is a good point, although randomly choosing a session is probably more complex asynchronously. One idea I just thought of (but I'm sure there is a better implementation) is:

1. Setup a task to handle the assignment of item objects) to a queue and it shuffles the list after each run through the list.
2. You set up a number of tasks equal to (*at least) the number of proxies you have available (*: I say at least, because this then limits the frequency checks if you have fewer proxies compared to items. Maybe it only works this way if you are using proxies? Otherwise it sets up number of tasks equal to items in list? Although that could get out of hand if someone has 100+ items and no proxies)
3. After check, if unsuccessful/no qualified sellers found, wait a delay period, plus some randomized amount (say 0-1 second) so the sessions do not keep their original positions in line.
4. If the check is successful, the monitoring task puts the qualified seller and item on to a "item found" queue. The ASIN assignment task pulls this off the queue, remembers the item that was found and puts the qualified seller into the checkout queue.
5. If the checkout was successful, return a message (via a different queue?) to the item assignment task and remove the item from the list.

I'm not super familiar with asyncio, so there might be better way to set up messages or do something with callbacks that could also achieve this functionality in a simpler way. I say this because the item assignment task is basically monitoring 3 different queues so I'm not sure if there is a way to await that, or if you need to loop through all of them continuously.

[hwamil]

Shit's broken so I'm going to start over with all the notes I've got. Thanks!