concurrent.futures._base.CancelledError when high throughput in aiohttp web application

[AnastagiZeno]

Hi there,

I am new in asyncio coding. Recently, I wrote a web application with aiohttp, during a request, I need do some db reading from and writing to cassandra. I use the aiocassandra as the driver, in the test environment, all stuff is ok. But in production environment, when the throughput(at the peek around 2k-3k QPS) is going high, I got these errors very frequently(50-80 per 15 minutes)

{'status': 'failed', 'content': None, 'errorCode': 'Internal Server Error', 'errorMessage': 'Traceback (most recent call last):\n  File "/app_libs/common/web/middlewares.py", line 74, in error_middleware\n    response = await handler(request)\n  File "/app_libs/common/web/middlewares.py", line 26, in request_trace_middleware\n    response = await handler(request)\n  File "/app_libs/common/web/middlewares.py", line 59, in track_time_middleware\n    response = await handler(request)\n  File "/app/routes/collecting.py", line 40, in webcollect\n    matchresult = await integration_data(request.app, request_data, config_data, cass_flag=True, kafka_flag=True)\n  File "/app/services/integration.py", line 34, in integration_data\n    result = await integration_event(app, request_data, configs, **kwargs)\n  File "/app/services/integration.py", line 62, in integration_event\n    user_data.online_data[\'page\'], app[\'cassandra\'], profile_id, user_data.device_id)\n  File "/app/services/integration.py", line 282, in check_pt_id\n    \'event\', \'device_id\', device_id, profile_id)\n  File "/app/libs/core.py", line 179, in cassandra_read\n    result.extend(await query(table, pk_str, pk_value))\n  File "/app/libs/core.py", line 168, in query\n    res = await session.execute_future(cql, (pk_v, ))\n  File "/usr/local/lib/python3.7/site-packages/aiocassandra.py", line 153, in execute_future\n    return await asyncio_fut\nconcurrent.futures._base.CancelledError\n'}

... execute_future\n    return await asyncio_fut\nconcurrent.futures._base.CancelledError\n'...

At first, I do those cassandra reading work after the request is finished, make those processes under an asynio task. Then when I got those errors, I wonder if that is caused by aiohttp framewok, as when a request is done, aiohttp may cancel tasks binded a request. Then I invoked aiojobs to make sure all those tasks would be waited even the response is returned. BUT the problem is still there, I try to put the cassandra operations into the request lifelong, take them out from the post-request background asyncio tasks, BUT, still, not solved the prolem. I search a lot on the internet, but not find anything really helpful.

I notice that this driver is wrapped with the datastax version with async query. I am not sure if I am using this driver correctlly or indeed there an issue exists. I dont know what to do with this, changing the thread/process executor or reuse the datastax non-aio verion back? Any advice from you guys?

Thank you very much. (By the way, I also use aiokafka in the same env but has not occured the same errors, and again, when the thoughput is at low level, all stuff is working.)

++++ python 3.7.4 aiohttp==3.6.0 aiocassandra==2.0.1 cassandra-driver==3.18.0

[ghost]

Do you have any solutions, I have the same problem here

[eavorobiev]

AFAIK, aiohttp throws CancelledError into coroutine if http connection was disconnected. So, the simplest solution is just to ignore this error (of course if you are not in distributed transaction etc etc). Anyway, it should not affect your request execution (because client has been disconnected already).

On Tue, 14 Jan 2020 at 09:49, elune notifications@github.com wrote:

Do you have any solutions, I have the same problem here

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aio-libs/aiocassandra/issues/108?email_source=notifications&email_token=AF7WEKAVYZBKONM4KGWDPM3Q5VOATA5CNFSM4IZGSECKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI3QANQ#issuecomment-574029878, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF7WEKCXDJDELJB7BBBNFILQ5VOATANCNFSM4IZGSECA .

-- Best regards, Evgeny Vorobiev

[ghost]

@eavorobiev Thanks for your reply, indeed this has something to do with connection disconnected

Thanks you very much