For some reason this process always goes only from 1st till 5th url, and then it just ignores the rest. Why is this happening? I'm absolutely sure, this is not because of URL inconsistency: i've tried to change order.
Also, I've created a queue to make all of the requests to be executed one-by-one. It works, all of the queued urls being scraped. But, when i'm comparing execution time even for 5 items in parallel and non-parallel, it shows, that queue stretches execution duration 4+ (!) times.
In fact, i need to run much more concurrent scraping processes. Is there way to make it work properly?
I'm looping through an array of urls (~12) with this script:
For some reason this process always goes only from 1st till 5th url, and then it just ignores the rest. Why is this happening? I'm absolutely sure, this is not because of URL inconsistency: i've tried to change order.
Also, I've created a queue to make all of the requests to be executed one-by-one. It works, all of the queued urls being scraped. But, when i'm comparing execution time even for 5 items in parallel and non-parallel, it shows, that queue stretches execution duration 4+ (!) times.
In fact, i need to run much more concurrent scraping processes. Is there way to make it work properly?