Remote Gremlin Server connections can be unexpectedly lost for a variety of reasons. As it currently stands, the AiohttpTransport implemented by gremlinpython does not really cope well with re-establishing connections. For long-lived Gremlin clients such as the one created by the nixpkgs-graph-explorer API this can be especially troublesome since the likelihood of a connection error increases with the longevity of the client.
A possible solution would be to extend the existing AiohttpTransport to attempt to reconnect when a failure is detected.
Initiating... (removing nodes and edges)
Adding nodes...
Traceback (most recent call last):
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 187, in <module>
main()
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 183, in main
ingest_graph(df, "ws://10.0.1.3:80/gremlin")
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 139, in ingest_graph
unique_insert_node(g, row)
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 81, in unique_insert_node
g.V().has("package", "outputPath", str(row["outputPath"])).fold().coalesce(
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 80, in iterate
try: self.nextTraverser()
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 88, in nextTraverser
return self.next_traverser()
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 92, in next_traverser
self.traversal_strategies.apply_strategies(self)
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 684, in apply_strategies
traversal_strategy.apply(traversal)
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/remote_connection.py", line 78, in apply
remote_traversal = self.remote_connection.submit(traversal.bytecode)
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/driver_remote_connection.py", line 105, in submit
results = result_set.all().result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/resultset.py", line 90, in cb
f.result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/connection.py", line 89, in _receive
data = self._transport.read()
File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/aiohttp/transport.py", line 106, in read
raise RuntimeError("Connection was already closed.")
RuntimeError: Connection was already closed.
Remote Gremlin Server connections can be unexpectedly lost for a variety of reasons. As it currently stands, the
AiohttpTransport
implemented bygremlinpython
does not really cope well with re-establishing connections. For long-lived Gremlin clients such as the one created by thenixpkgs-graph-explorer
API this can be especially troublesome since the likelihood of a connection error increases with the longevity of the client.A possible solution would be to extend the existing
AiohttpTransport
to attempt to reconnect when a failure is detected.Related StackOverflow: https://stackoverflow.com/questions/70109681/neptune-runtimeerror-connection-was-already-closed-in-pythongremlin
Example traceback (from
etl.py
in this case):