neo4j-contrib / neomodel

An Object Graph Mapper (OGM) for the Neo4j graph database.
https://neomodel.readthedocs.io
MIT License
939 stars 231 forks source link

Explicit closing of the driver #697

Closed dzianis-hrynko closed 1 year ago

dzianis-hrynko commented 1 year ago

Expected Behavior (Mandatory)

Saving prepared entity without any exceptions

Actual Behavior (Mandatory)

Entities have been saved in DB, however the exceptions has thrown

` Exception ignored in: <function Driver.del at 0x7f1fe649e440> Traceback (most recent call last): File ".local/lib/python3.10/site-packages/neo4j/_sync/driver.py", line 411, in del File ".local/lib/python3.10/site-packages/neo4j/_sync/driver.py", line 456, in close File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_pool.py", line 367, in close File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_pool.py", line 320, in _close_connections File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 757, in close File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 609, in _send_all File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_common.py", line 145, in flush File ".local/lib/python3.10/site-packages/neo4j/_async_compat/util.py", line 113, in callback File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 688, in _set_defunct_write File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 691, in _set_defunct ImportError: sys.meta_path is None, Python is likely shutting down Exception ignored in: <function Bolt.del at 0x7f1fe64679a0> Traceback (most recent call last): File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 161, in del File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 757, in close File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 609, in _send_all File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_common.py", line 145, in flush File ".local/lib/python3.10/site-packages/neo4j/_async_compat/util.py", line 113, in callback File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 688, in _set_defunct_write File ".local/lib/python3.10/site-packages/neo4j/_sync/io/_bolt.py", line 691, in _set_defunct ImportError: sys.meta_path is None, Python is likely shutting down

Process finished with exit code 0 `

How to Reproduce the Problem

I use an example of the code from documentation https://neomodel.readthedocs.io/en/latest/getting_started.html

Simple Example

` from neomodel import (config, StructuredNode, StringProperty, IntegerProperty, UniqueIdProperty)

user = 'your-user' psw = 'your-pass' uri = 'your-db-id.databases.neo4j.io'

config.DATABASE_URL = 'neo4j+s://{}:{}@{}'.format(user, psw, uri)

class Person(StructuredNode): uid = UniqueIdProperty() name = StringProperty(unique_index=True) age = IntegerProperty(index=True, default=0)

if name == "main": jim = Person(name='Jisdrm', age=3).save() # Create jim.name = "adasd" jim.save() # Update, (with validation)

`

Screenshots (where it's possibile)

image

Specifications (Mandatory)

5.0.0

Versions

aanastasiou commented 1 year ago

@dzianis-hrynko Thanks for raising this with us

To my understanding this is an exception that was thrown at the level of the neo4j python3 driver rather than the level of neomodel (or as a result of a neomodel call).

Most probably because your program run very fast locally and exited before the driver had finished its own "destruction". Can you please share a bit more of your code from if __name__ == "__main__": onwards?

Also, please refrain from posting credentials and full URLs in the future when posting examples such as this. The usernames, passwords and connection URL are not important to understand what the issue is about here.

dzianis-hrynko commented 1 year ago

@aanastasiou

Can you please share a bit more of your code from if name == "main": onwards?

That's all I have in if __name__ == "__main__": image

I've tested how neomodel works to choose neomodel or just python driver use in my projects and faced with that issues.

Most probably because your program run very fast locally and exited before the driver had finished its own "destruction". It's weird. Anyway, I've added sleep after saving to give time to destruct, and the exception is the same.

Maybe there is another way to solve the issues? I'm not sure if it's possible to destruct the driver forcibly

aanastasiou commented 1 year ago

@dzianis-hrynko You can find a little bit more on how to choose here.

I will try to have a closer look into this, to try and replicate the error, but I am positive that it stems from the driver, not neomodel. You might have your question answered much faster if you ask over at the driver's repository space.

mariusconjeaud commented 1 year ago

Hello @dzianis-hrynko can you with release 5.1.0 ? It updates the driver to latest 5.10

navindev26 commented 1 year ago

I had the same issue when using Neo4jGraph via LangChain. The script seems to finish before the database driver completes its cleanup. Fixed it by calling close() explicitly at the end of

if __name__ == "__main__":

uri = "neo4j://example.com:7687"
driver = GraphDatabase.driver(uri, auth=("neo4j", "password"))
driver.close()  # close the driver object.

for LangChain using the wrapper Neo4jGraph it was neo_graph._driver.close()

mariusconjeaud commented 1 year ago

This is a general issue that we are facing and thinking about, because until version 5.x of the Neo4j Python driver, it would close itself, but it is now deprecated in version 5.x This means we get quite a few deprecation warnings when running the tests about this.

I just couldn't figure out the best place to explicitly close the driver yet though. Maybe the solution lies in providing a method to call to the users, so that they can close the driver at the end of their app's lifecycle. It would also become irrelevant if we move the driver creation out of neomodel and into the user's hands as the other OGMs do (like SDN) ; then the lifecycle of the driver is managed by the app and not the library, which makes more sense.

After chatting with the Neo4j driver team, there is no single good solution for this ; but on the other hand, it is also not critical and doesn't make neomodel unfunctional.

So we can safely ignore this and focus on the question of extracting the driver outside of neomodel's scope, which would solve this in a more robust way.