RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x55dc68caf9f0>; the handl...

[sentry-io[bot]]

Sentry Issue: OPENVERSE-API-YE

RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x55dc68caf9f0>; the handler is closed
  File "uvloop/sslproto.pyx", line 714, in uvloop.loop.SSLProtocol._do_read
  File "uvloop/sslproto.pyx", line 699, in uvloop.loop.SSLProtocol._process_outgoing
  File "uvloop/handles/stream.pyx", line 674, in uvloop.loop.UVStream.write
    self._ensure_alive()
  File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
    raise RuntimeError(

Fatal error on SSL protocol
protocol: <uvloop.loop.SSLProtocol object at 0x7fbb6a8681a0>
transport: <TCPTransport closed=True reading=False 0x55dc68caf9f0>

[sarayourfriend]

I thought this would be fixed in https://github.com/WordPress/openverse/pull/4875, but they are still happening.

This error is worrisome because there is very little visibility into what the effect of it is, and it's been difficult to understand exactly where it comes from, what causes it, or even when it happens.

My suggested plan of action to try to address this is, unfortunately, purely speculative. It is as follows:

• Upgrade aiohttp to make sure this isn't a bug fix with uvloop compatibility
  • I somewhat doubt this will fix it, but it's the easiest first-thing to try
• Find out if there is a way to check for closed TCPTransports in get_aiohttp_session, if for some reason checking that the session is closed isn't doing the trick
  • I've looked through aiohttp's code a lot, and I've convinced myself this isn't possible... but I was looking at the version of aiohttp we currently use, and there are aiohttp changelog references to transport improvements, so things might be different after we upgrade aiohttp.
  • If we can't directly check if the TCPTransport is closed, maybe there are some aiohttp tracing hooks where we could do that, and through that potentially find out at least the context this happens in...
• Upgrade to Django 5.1 so we can take advantage of the cancelled request (on the client side) handling, which may be causing some other cleanup issues
• Try switching to HTTPx instead of aiohttp (pook supports both now, so it should be possible); there are indications in the GitHub issues for uvloop, uvicorn, httpx, aiohttp, and asgiref that aiohttp may have issues with uvloop, even if not fixed in upgrading
• Try running uvicorn with non-uvloop and see if it goes away (which would indicate a uvloop-related issue rather than something else we've done wrong or some other library)

[sarayourfriend]

@WordPress/openverse-api is anyone interested in taking this on? I've recommended a list of actions to try in the previous comment. I ordered them based on my estimation of how much effort they would take, and I would suggest trying them in that order unless you disagree with my relative estimations. After each one, we would need to deploy to production and see whether the RuntimeError is still happening.

Please ping me on any PRs that open as a result of this issue. I think someone in @WordPress/openverse-api should drive the work itself so that the understanding of it is within that group of maintainers, but I am happy to advise, review, and lend any other support to anyone working through this.