search

neo4j / neo4j-python-driver

Neo4j Bolt driver for Python
https://neo4j.com/docs/api/python-driver/current/
Other
909 stars 186 forks source link

transaction.commit() failures on larger data volumes #377

Closed jholtom-ciena closed 4 years ago

jholtom-ciena commented 4 years ago

Neo4j Version: 4.0.1 Enterprise & 3.5.14 Enterprise (Desktop) Neo4j Mode: Single instance
Driver version: Python driver 4.0.dev0 Operating System: macOS 10.15.3

Steps to reproduce

1) Start Neo4j (4.0 or 3.5) with a clean empty database 2) Run script

from neo4j import GraphDatabase,basic_auth
import neo4j

scale=340

if '1.7.6' == neo4j.__version__:
    driver = GraphDatabase.driver('bolt://localhost:7687', encrypted=False, auth=basic_auth('neo4j','test'))
else:
    driver = GraphDatabase.driver('bolt://localhost:7687', auth=basic_auth('neo4j','test'))

session= driver.session()

tx=session.begin_transaction()
mkNode = "CREATE (e:Node {nodeId:$rand,name:$name})"
for num in range (0,scale):
    tx.run(mkNode,{'rand':num,'name':'Node-'+str(num)})

tx.commit()

mkMesh = """
WITH $props AS props
MATCH (a:Node {nodeId:$thingA}), (b:Node {nodeId:$thingB}) 
MERGE (a)-[re:RELATES {relatesId:$thingA-$thingB}]->(b)
    ON CREATE SET re+=props
MERGE (r:Relation {relationId:$thingA-$thingB})
    ON CREATE SET r+=props
MERGE (a)<-[:RELATES]-(r)-[:RELATES]->(b)
"""

props = {"StringField-000001":"1234","StringField-000002":"2020-03-20","StringField-000003":1234,"EmptyField-000001":"","StringField-000004":42,"StringField-000005":"Abcdefg","StringField-000006":"Ghijklmn","StringField-000007":"AbcdefghijklmnopqrstuvwxyzAbcdefghijklmnopqrstuvwxyz","StringField-000008":"AbcdefghijklmnopqrstuvwxyzAbcdefghijklmnopqrstuvwxyz","StringField-000009":"123","StringField-000010":42,"StringField-000011":"Abcde1","StringField-000012":"aBCDE1","StringField-000013":"Zyxwv2","StringField-000014":"zYXWV2","StringField-000015":"ABCDWXYZ","StringField-000016":"4","StringField-000017":"zyxwvutsrqponmlkjihgfedcba","StringField-000018":"2","StringField-000019":"1","StringField-000020":"3","StringField-000021":"2","StringField-000022":"1","StringField-000023":"4","StringField-000024":"abcdefghijklmnopqrstuvwxyz","StringField-000025":"5","StringField-000026":"2"}

i=0
tx=session.begin_transaction()
for thingA in range (0,scale):
    for thingB in range (0,scale):
        if thingA != thingB:
            tx.run(mkMesh,{'thingA':thingA,'thingB':thingB,'props':props})
            i+=1
            print('{:6d}: Node-{:3d}=Node-{:3d}'.format(i,thingA,thingB))
        if divmod(i,1000)[1] == 0:
            print("Commit-point: "+str(i))
            tx.commit()
            tx=session.begin_transaction()

tx.commit()

Expected behaviour

Script should run to completion, mirroring behaviour when using the neo4j==1.7.6 driver stack

Actual behaviour

The commit phase causes an exception:

With a v4.0 database:

      [deletia]

   999: Node-  2=Node-321
  1000: Node-  2=Node-322
Commit-point: 1000
Failed to write data to connection IPv4Address(('localhost', 7687)) (IPv4Address(('127.0.0.1', 7687))); ("35; 'Resource temporarily unavailable'")
Failed to write data to connection IPv4Address(('localhost', 7687)) (IPv4Address(('127.0.0.1', 7687))); ("35; 'Resource temporarily unavailable'")
Traceback (most recent call last):
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 557, in _close
    self.sync()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 530, in sync
    self.session.sync()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 247, in sync
    self._connection.send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt4x0.py", line 268, in send_all
    self._send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt4x0.py", line 253, in _send_all
    self.socket.sendall(data)
BlockingIOError: [Errno 35] Resource temporarily unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "testloader.py", line 42, in <module>
    tx.commit()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 539, in commit
    self._close()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 564, in _close
    self.session.commit_transaction()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 329, in commit_transaction
    self._connection.send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt4x0.py", line 268, in send_all
    self._send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt4x0.py", line 253, in _send_all
    self.socket.sendall(data)
BlockingIOError: [Errno 35] Resource temporarily unavailable

With a v3.5 database:

   998: Node-  2=Node-320
   999: Node-  2=Node-321
  1000: Node-  2=Node-322
Commit-point: 1000
Failed to write data to connection IPv4Address(('localhost', 7687)) (IPv4Address(('127.0.0.1', 7687))); ("35; 'Resource temporarily unavailable'")
Failed to write data to connection IPv4Address(('localhost', 7687)) (IPv4Address(('127.0.0.1', 7687))); ("35; 'Resource temporarily unavailable'")
Traceback (most recent call last):
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 557, in _close
    self.sync()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 530, in sync
    self.session.sync()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 247, in sync
    self._connection.send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt3.py", line 270, in send_all
    self._send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt3.py", line 255, in _send_all
    self.socket.sendall(data)
BlockingIOError: [Errno 35] Resource temporarily unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "testloader.py", line 42, in <module>
    tx.commit()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 539, in commit
    self._close()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 564, in _close
    self.session.commit_transaction()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/work/simple.py", line 329, in commit_transaction
    self._connection.send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt3.py", line 270, in send_all
    self._send_all()
  File "/Users/jholtom/playpen/py3env/lib/python3.7/site-packages/neo4j-4.0.dev0-py3.7.egg/neo4j/io/_bolt3.py", line 255, in _send_all
    self.socket.sendall(data)
BlockingIOError: [Errno 35] Resource temporarily unavailable

Despite enabling DEBUG level logging, no errors are reported from the server

technige commented 4 years ago

@jholtom-ciena Which version of Python are you running?

jholtom-ciena commented 4 years ago

Sorry, forgot that:

(py3env) playpen % python -V
Python 3.7.6
technige commented 4 years ago

Fixed by https://github.com/neo4j/neo4j-python-driver/pull/378