Added to new command line args:
--batch-size is intended to be an int between 1 and 100. This dictates the batch size of urls we are trying to process at once. When getting errors reducing this number has been helpful while slowing down the speed of the script.
--semaphore-size is intended to be an int between 1 and 50. From the batch above, this number determines how many urls we try to process asynchronously at a given time. Reducing this number when receiving errors seems to have a positive effect. While also slowing down the speed of the script.
Added to new command line args:
--batch-sizeis intended to be an int between 1 and 100. This dictates the batch size of urls we are trying to process at once. When getting errors reducing this number has been helpful while slowing down the speed of the script.--semaphore-sizeis intended to be an int between 1 and 50. From the batch above, this number determines how many urls we try to process asynchronously at a given time. Reducing this number when receiving errors seems to have a positive effect. While also slowing down the speed of the script.