Closed williamhakim10 closed 3 months ago
Hi and thanks for reaching out.
If my analysis is correct, the problem stems from the line
asyncio.run(self.__async_driver.close())
Let me quote the docs of asyncio.run
:
This function runs the passed coroutine, taking care of managing the asyncio event loop, finalizing asynchronous generators, and closing the executor.
This function cannot be called when another asyncio event loop is running in the same thread.
So a new event loop will be started to execute the passed coroutine (driver.close()
). With that in mind the error you're seeing
[...] Task-364892 [...] got Future [...] attached to a different loop might make more sense. The moment you first use the driver to execute async work, it will be bound to the event loop executing the work. I/O streams, locks, etc., all assume they belong to a singe event loop.
I'll close the issue for now as it does not appear to be an issue with the driver. But please feel free to keep commenting if you have further questions.
Hi @robsdedude I think you're correct in your analysis, but why does it work that way? The various other async services which we run (sqlalchemy, redis, etc.) are totally fine with being torn down in a different event loop from the one in which they do the work.
I understand the ideal refactor here is to set up and tear down the inside a single block of async code? But that pattern basically assumes that your code is mostly or entirely async, rather than writing mostly sync code which occasionally does async stuff.
The various other async services which we run (sqlalchemy, redis, etc.) are totally fine with being torn down in a different event loop from the one in which they do the work.
Here, I can only speculate, as I don't know the internals of the libraries. Maybe they are doing less/no async work at teardown. They could, for example just close the sockets without waiting. Whereas this driver tries to be a good citizen by sending a GOODBYE
message to the server before shutting down the socket.
I understand the ideal refactor here is to set up and tear down the inside a single block of async code?
Yes, unless you manage the eventloop manually and pass it around. However, I see your code uses both, an async and a sync driver. But while you're running inside a coroutine (or async Task, etc.) you shouldn't be using any blocking operations like those the sync driver performs as that would block the event loop preventing other async tasks from progressing. As you wrote: mixing async and sync code is an error prone approach in general.
Yeah, definitely not doing any blocking code inside a coroutine, so that's not the concern.
The reason for declaring both a synchronous and asynchronous driver is so that configuration can happen at the app's entrypoint, rather than pushing configuration/environment variables/etc. down into random blocks of async code which may live in shared libraries or other places that don't know anything about their environment. It sounds like the only way to get around this is to manually pass around the event loop somehow?
Bug Report
We've been experiencing issues with event loops and trying to
close()
on anAsyncNeo4jDriver
.This is the error we've been seeing:
Our code for handling graph db drivers looks like this. I am including the handlers for dead connections for completeness, but I've tried removing them and refactoring the code to use
execute_query
or something else instead, and it doesn't matter.This code gets called inside of a Cloud Function which looks like this:
I have tried various permutations of keeping track of the event loop (i.e. by storing it in the template) and then using a lower-level asyncio primitive which allows passing in a loop, as well as just allowing the driver to clean itself up without doing anything at all - in both cases I get a
RuntimeError('Event loop is closed')
.Note that I am able to recreate this behavior using a simple unittest test case with only the following parameters as long as the test in question actually queries the graph.
I see that this came up previously (https://github.com/neo4j/neo4j-python-driver/issues/950) but obviously we're on 3.10 so that particular issue hasn't gone away.
Thanks for taking a look and appreciate the support!
My Environment
Python Version: Python 3.10.13 (main, Apr 8 2024, 10:10:13) [Clang 15.0.0 (clang-1500.3.9.4)] Driver Version: 5.19.0 Server Version and Edition: Neo4j EE 5.18.0 (locally, in tests) but also AuraDB Operating System: MacOS Sonoma 14.3, but also
ubuntu:latest
in Github CI and whatever linux version Cloud Run uses.