Closed cgoma closed 7 months ago
You need to understand that when you use asyncio.create_task
, that task is started immediately, and asyncio.gather
is only used to wait for a specific task.
So you started 5000 tasks at the same time and in your for task in chunk_list
loop you're waiting for each of those tasks in sequence, but they all already started.
So you're running into a timeout which is implemented in the aiohttp library.
By default there is a 5 minutes timeout. See https://docs.aiohttp.org/en/stable/client_quickstart.html#timeouts
You could probably modify that timeout by passing a custom aiohttp.ClientTimeout
object to the timeout
key of the client_session_args
parameter of AIOHTTPTransport
, but a better way to fix this would be just to reorganize your code to maybe have a pool of 10 asyncio tasks waiting for new jobs to do from a asyncio queue, or at least create your tasks 10 by 10 instead of creating them all at once.
Describe the problem When number of tasks which executes mutation goes beyond 5000+ , timeout error occurred for some of the tasks. When the task number is less i.e. around 300, all mutation executed successfully. For 5000 tasks as well, it succeeded few times but for most of the time it failed.
I want to make sure all the tasks (5000+) should be successful. Not sure why it worked some time and fails most of the time.
The code which I am using is as below. How I can make sure all the 5000 mutation should be executed successfully without fail
Below are the exceptions I am getting