RuntimeError in middleware

The following exception was logged

Traceback (most recent call last):
File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/middleware.py", line 46, in exception_handling_middleware
    return await handler(request)
File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/remote.py", line 16, in middleware
    return await super().middleware(request, handler)
File "/var/www/eidaws-federator/venv/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 94, in middleware
    return await handler(request)
File "/var/www/eidaws-federator/venv/lib/python3.7/site-packages/aiohttp/web_urldispatcher.py", line 948, in _iter
    resp = await method()
File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/view.py", line 52, in get
    return await processor.federate(timeout=self.client_timeout)
File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/process.py", line 104, in wrapper
    return await coro(self, *args, **kwargs)
    RuntimeError: coroutine ignored GeneratorExit

which originally referred to the request (aiohttp access log):

2021-04-13T06:06:13.684492 "GET /fedws/wfcatalog/json/1/query?starttime=2004-01-01T00:00:00&endtime=2004-01-02T00:00:00&net=CH HTTP/1.1' '-' 'ApacheBench/2.3'

Logs:

Apr 13 06:06:13 a6e70dd05f2a <EIDA> 2021-04-13T06:06:13+0000 INFO eidaws.federator.middleware 75 misc.py:229 - [3ae057cd-8baa-45d7-b449-b7d991ab200f]  2021-04-13T06:06:13.684492 "GET /fedws/wfcatalog/json/1/query?starttime=2004-01-01T00:00:00&endtime=2004-01-02T00:00:00&net=CH&starttime=2004-01-01T00:00:00&endtime=200
4-01-02T00:00:00&net=CH HTTP/1.1' '-' 'ApacheBench/2.3'
[...]
Apr 13 07:34:56 a6e70dd05f2a <EIDA> 2021-04-13T07:34:56+0000 CRITICAL eidaws.federator.middleware 75 middleware.py:66 - [3ae057cd-8baa-45d7-b449-b7d991ab200f] Local Exception: <class 'RuntimeError'>
Apr 13 07:34:56 a6e70dd05f2a <EIDA> 2021-04-13T07:34:56+0000 CRITICAL eidaws.federator.middleware 75 middleware.py:70 - [3ae057cd-8baa-45d7-b449-b7d991ab200f] Traceback information: ['Traceback (most recent call last):\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/middleware.py", line 46, in exception_handling_middleware\n    return await handler(request)\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/remote.py", line 16, in middleware\n    return await super().middleware(request, handler)\n', '  File "/var/www/eidaws-federator/venv/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 94, in middleware\n    return await handler(request)\n', '  File "/var/www/eidaws-federator/venv/lib/python3.7/site-packages/aiohttp/web_urldispatcher.py", line 948, in _iter\n    resp = await method()\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/view.py", line 52, in get\n    return await processor.federate(timeout=self.client_timeout)\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/process.py", line 104, in wrapper\n    return await coro(self, *args, **kwargs)\n', 'RuntimeError: coroutine ignored GeneratorExit\n']

Note the timestamps of the log messages.

The issue seems to be reproducible, however it seems to be always related to eidaws-wfcatalog requests.

Dependencies:

# pip freeze
aiodns==2.0.0
aiofiles==0.6.0
aiohttp==3.7.4.post0
aiohttp-cors==0.7.0
aiohttp-remotes==1.0.0
aioredis==1.3.1
async-timeout==3.0.1
attrs==20.3.0
brotlipy==0.7.0
cached-property==1.5.2
cchardet==2.1.7
cffi==1.14.5
chardet==4.0.0
ConfigArgParse==1.4
# Editable install with no version control (eidaws.federator==0.12.0)
-e /usr/local/src/eidaws/eidaws.federator
# Editable install with no version control (eidaws.utils==0.1)
-e /usr/local/src/eidaws/eidaws.utils
hiredis==2.0.0
idna==3.1
importlib-metadata==3.0.0
intervaltree==3.1.0
jsonschema==3.2.0
lxml==4.6.3
marshmallow==3.2.1
multidict==4.7.6
pkg-resources==0.0.0
pycares==3.1.1
pycparser==2.20
pyrsistent==0.17.3
python-dateutil==2.8.1
PyYAML==5.4.1
six==1.15.0
sortedcontainers==2.3.0
typing-extensions==3.7.4.3
webargs==5.5.3
yarl==1.5.1
zipp==3.4.1

The issue seems to be occur also for fdsnws-station-text requests.

Logs:

Apr 19 18:14:04 a478bf71e411 <EIDA> 2021-04-19T18:14:04+0000 INFO eidaws.federator.middleware 94 misc.py:229 - [a3342724-fef4-4a95-b530-9007aa4a2d97]  2021-04-19T18:14:04.167155 "GET /fedws/station/text/1/query?format=text&level=station&network=DK&station=ICESG&location=*&channel=*&starttime=2011-06-09T00:00:00.000001  HTTP/1.1' '-' 'EIDA-Crawler/0.12.0'
Apr 19 18:14:05 a478bf71e411 <EIDA> 2021-04-19T18:14:05+0000 CRITICAL eidaws.federator.middleware 94 middleware.py:66 - [a3342724-fef4-4a95-b530-9007aa4a2d97] Local Exception: <class 'RuntimeError'>
Apr 19 18:14:05 a478bf71e411 <EIDA> 2021-04-19T18:14:05+0000 CRITICAL eidaws.federator.middleware 94 middleware.py:70 - [a3342724-fef4-4a95-b530-9007aa4a2d97] Traceback information: ['Traceback (most recent call last):\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/middleware.py", line 46, in exception_handling_middleware\n    return await handler(request)\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/remote.py", line 16, in middleware\n    return await super().middleware(request, handler)\n', '  File "/var/www/eidaws-federator/venv/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 94, in middleware\n    return await handler(request)\n', '  File "/var/www/eidaws-federator/venv/lib/python3.7/site-packages/aiohttp/web_urldispatcher.py", line 948, in _iter\n    resp = await method()\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/view.py", line 52, in get\n    return await processor.federate(timeout=self.client_timeout)\n', '  File "/usr/local/src/eidaws/eidaws.federator/eidaws/federator/utils/process.py", line 104, in wrapper\n    return await coro(self, *args, **kwargs)\n', 'RuntimeError: coroutine ignored GeneratorExit\n']

EIDA / eidaws

RuntimeError in middleware #9