search

pulp / pulpcore

Pulp 3 pulpcore package https://pypi.org/project/pulpcore/
GNU General Public License v2.0
284 stars 112 forks source link

Occasional `Connection reset by peer` by content app #2334

Open goosemania opened 2 years ago

goosemania commented 2 years ago

Occasionally the pulp-content-app will return a 500 error due to a Connection reset by peer error. Subsequent requests complete successfully.

It has been observed that once this occurs it will occur repeatedly albeit infrequently until the pod is deleted/restarted. Additionally the error typically occurs in only one of the three pulp-content-app replicas at a time.

[2022-02-24 19:10:18 +0000] [10] [ERROR] Error handling request
Traceback (most recent call last):
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/venv/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/venv/lib64/python3.8/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
    return await handler(request)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 212, in cached_function
    bk = await self.default_base_key(request, self)
  File "/venv/lib64/python3.8/site-packages/pulpcore/content/handler.py", line 156, in find_base_path_cached
    index_p1 = await cached.exists(base_key=multiplied_base_paths)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 50, in wrapper
    return await func(*args, **kwargs)
  File "/venv/lib64/python3.8/site-packages/pulpcore/cache/cache.py", line 153, in exists
    return await self.redis.exists(*base_key)
  File "/venv/lib64/python3.8/site-packages/aioredis/client.py", line 1061, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 1365, in get_connection
    if await connection.can_read():
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 848, in can_read
    return await self._parser.can_read(timeout)
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 362, in can_read
    return self._buffer and bool(await self._buffer.can_read(timeout))
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 274, in can_read
    return bool(self.length) or await self._read_from_socket(
  File "/venv/lib64/python3.8/site-packages/aioredis/connection.py", line 247, in _read_from_socket
    data = await self._stream.read(self.socket_read_size)
  File "/usr/lib64/python3.8/asyncio/streams.py", line 665, in read
    raise self._exception
  File "/usr/lib64/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
2022-02-24 19:10:18,043 INFO gunicorn.access: X.X.X.X [24/Feb/2022:19:10:18 +0000] "GET /api/automation-hub/v3/artifacts/collections/published/ansible-tower-3.8.5.tar.gz?validate_token=539ad80e030608fa60520eb76a9553050727b4a2b95889b7de48ab81553be496:56be102481103ad85a7d0c4ec8b762a86767cdd21e84186e717b8419966014cb HTTP/1.1" 500 244 "-" "python-requests/2.26.0"

pulpcore==3.17.4 aioredis==2.0.0 AWS Elasticache for Redis: 5.0.5

goosemania commented 2 years ago

ipanova: it can be this issue https://github.com/aio-libs/aioredis-py/issues/778