search

tweag / nixpkgs-graph-explorer

Explore the nixpkgs dependency graph
MIT License
15 stars 0 forks source link

Improve logic for retrying client connections on disconnect #72

Closed dorranh closed 1 year ago

dorranh commented 1 year ago

Remote Gremlin Server connections can be unexpectedly lost for a variety of reasons. As it currently stands, the AiohttpTransport implemented by gremlinpython does not really cope well with re-establishing connections. For long-lived Gremlin clients such as the one created by the nixpkgs-graph-explorer API this can be especially troublesome since the likelihood of a connection error increases with the longevity of the client.

A possible solution would be to extend the existing AiohttpTransport to attempt to reconnect when a failure is detected.

Related StackOverflow: https://stackoverflow.com/questions/70109681/neptune-runtimeerror-connection-was-already-closed-in-pythongremlin

Example traceback (from etl.py in this case):

Initiating... (removing nodes and edges)
Adding nodes...
Traceback (most recent call last):
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 187, in <module>
    main()
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 183, in main
    ingest_graph(df, "ws://10.0.1.3:80/gremlin")
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 139, in ingest_graph
    unique_insert_node(g, row)
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/etl.py", line 81, in unique_insert_node
    g.V().has("package", "outputPath", str(row["outputPath"])).fold().coalesce(
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 80, in iterate
    try: self.nextTraverser()
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 88, in nextTraverser
    return self.next_traverser()
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 92, in next_traverser
    self.traversal_strategies.apply_strategies(self)
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/process/traversal.py", line 684, in apply_strategies
    traversal_strategy.apply(traversal)
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/remote_connection.py", line 78, in apply
    remote_traversal = self.remote_connection.submit(traversal.bytecode)
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/driver_remote_connection.py", line 105, in submit
    results = result_set.all().result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/resultset.py", line 90, in cb
    f.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/connection.py", line 89, in _receive
    data = self._transport.read()
  File "/home/dorran_howell/nixpkgs-graph-explorer/etl/venv/lib/python3.9/site-packages/gremlin_python/driver/aiohttp/transport.py", line 106, in read
    raise RuntimeError("Connection was already closed.")
RuntimeError: Connection was already closed.