Error: while trying to create huge network

Trying to create a network of 50000+ Nodes and 100000+ Relationships.

In mid way getting the following error (Around 53900 Relationship create point):

`Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/py2neo/client/init.py", line 792, in acquire cx = self._free_list.popleft() IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "neo4j_converter.py", line 273, in main() File "neo4j_converter.py", line 267, in main create_relationship(relationships) File "neo4j_converter.py", line 201, in create_relationship g.create(Relationship(a, c, b)) File "/usr/local/lib/python3.8/dist-packages/py2neo/database.py", line 580, in create self.update(lambda tx: tx.create(subgraph)) File "/usr/local/lib/python3.8/dist-packages/py2neo/database.py", line 439, in update self._update(cypher, timeout=timeout) File "/usr/local/lib/python3.8/dist-packages/py2neo/database.py", line 461, in _update tx = self.begin( File "/usr/local/lib/python3.8/dist-packages/py2neo/database.py", line 351, in begin return Transaction(self, autocommit=False, readonly=readonly, File "/usr/local/lib/python3.8/dist-packages/py2neo/database.py", line 904, in init self._ref = self._connector.begin(self.graph.name, readonly=readonly, File "/usr/local/lib/python3.8/dist-packages/py2neo/client/init.py", line 1369, in begin cx = self._acquire_rw(graph_name) File "/usr/local/lib/python3.8/dist-packages/py2neo/client/init.py", line 1240, in _acquire_rw cx = pool.acquire() File "/usr/local/lib/python3.8/dist-packages/py2neo/client/init.py", line 799, in acquire cx = self.connect() File "/usr/local/lib/python3.8/dist-packages/py2neo/client/init.py", line 745, in connect self._bytes_sent += cx.bytes_sent File "/usr/local/lib/python3.8/dist-packages/py2neo/client/init.py", line 211, in bytes_sent raise NotImplementedError NotImplementedError `

What could be the possible reason for this?

Hello,

I am having the same problem. I am trying to create over 14m relationships using apoc.periodic.iterate procedure. I have 19 separate queries, which are sequentially executed in the same Python method using run method from Graph.

However, at the 17th query it throws the following error (previous queries take approximately ~3739 seconds):

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/py2neo/client/__init__.py", line 792, in acquire
    cx = self._free_list.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 731, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/utility/tasks.py", line 83, in run_queries
    execute_queries(queries, task_id, user_id)
  File "/app/utility/tasks.py", line 24, in execute_queries
    graph_access.batch(query[0], query[1], params=params)
  File "/usr/local/lib/python3.9/site-packages/context_sense/graph_access.py", line 98, in batch
    return self.run(query)
  File "/usr/local/lib/python3.9/site-packages/context_sense/graph_access.py", line 120, in run
    result = self.graph.run(query, params).data()
  File "/usr/local/lib/python3.9/site-packages/py2neo/database.py", line 405, in run
    return self.auto().run(cypher, parameters, **kwparameters)
  File "/usr/local/lib/python3.9/site-packages/py2neo/database.py", line 978, in run
    result = self._connector.auto_run(cypher, parameters,
  File "/usr/local/lib/python3.9/site-packages/py2neo/client/__init__.py", line 1338, in auto_run
    cx = self._acquire_rw(graph_name)
  File "/usr/local/lib/python3.9/site-packages/py2neo/client/__init__.py", line 1240, in _acquire_rw
    cx = pool.acquire()
  File "/usr/local/lib/python3.9/site-packages/py2neo/client/__init__.py", line 799, in acquire
    cx = self.connect()
  File "/usr/local/lib/python3.9/site-packages/py2neo/client/__init__.py", line 745, in connect
    self._bytes_sent += cx.bytes_sent
  File "/usr/local/lib/python3.9/site-packages/py2neo/client/__init__.py", line 211, in bytes_sent
    raise NotImplementedError
NotImplementedError

I have no error on the Neo4j server logs. So, I guess it is a problem on the py2neo client side.

What could be the reason:

Does the connections in connection pool expire? Can I check to see if connections are expired and reconnect somehow?
Do I need to ditch run and execute each query within a separate transaction (this doesn't make sense to me, because I think run uses auto() to create a separate transaction for queries)?

By the way, what is the best practice to use Graph objects, which of the following is better:

to initialize a Graph object before executing any query?
to create a Graph object once when an application initialized and use this object during the lifetime of the application?

neo4j-contrib / py2neo

Error: while trying to create huge network #906