graphcore / poptorch

PyTorch interface for the IPU
https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/
MIT License
177 stars 15 forks source link

Fabric error: connection not established #14

Open Lime-Cakes opened 1 year ago

Lime-Cakes commented 1 year ago

I got the following out of the blue while testing some code on free pod4 machines. After restarting machine, the same loading code seemed to run fine, being stable and produce inferenceModel that can be run and produce correct results.

  File "<ipython-input-10-d80f14a13956>", line 54, in restoreSavedModel
    inference_model.loadExecutable(
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_impl.py", line 358, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py", line 1000, in loadExecutable
    self._on_device_attach()
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py", line 383, in _on_device_attach
    self.copyWeightsToDevice()
  File "/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py", line 562, in copyWeightsToDevice
    poptorch_core.copyWeightsToDevice_impl(self._executable,
poptorch.poptorch_core.UnrecoverableError: In unknown:0: 'poplar_unrecoverable_runtime_error': fabric_atomic_increment_hsp_register: Could not complete operation. Fabric error: connection not established: CONNECTION_KILL, device 3, IP 10.5.17.11:50052. IPUoF Server unreachable at 2023-02-03T15:32:48.181819Z
Error raised in:
  [0] copyWeightsToDevice
payoto commented 1 year ago

Hello @Lime-Cakes, This issue is due to a problem with the machine. I will reach out to Paperspace Gradient to investigate which machine is affected but it would be very useful if you could contact their support with this error and provide them with the "notebook ID" if you still have it. The "notebook ID" can be found in your list of notebooks:

Screenshot 2023-02-06 at 12 10 50

Thanks for reporting this error, Alexandre