search

graphcore / Gradient-Pytorch-Geometric

A repository of tutorials and examples demonstrating use of PyTorch Geometric with IPUs
MIT License
6 stars 1 forks source link

learning-pytorch-geometric-on-ipus/ notebooks have C++ endpoint not connected errors #9

Open nmb-paperspace opened 1 year ago

nmb-paperspace commented 1 year ago

2_a_worked_example.ipynb cell 11 from tqdm import tqdm ... fails with

Error: In unknown:0: 'std::exception': boost::filesystem::directory_iterator::operator++: Transport endpoint is not connected: "/tmp/exe_cache/pyg-a-worked-example"
Error raised in:
  [0] popart::TrainingSession::createFromOnnxModel
  [1] Compiler::initSession
  [2] LowerToPopart::compile
  [3] compileWithManualTracing

3_small_graph_batching_with_padding.ipynb cell 14 fails with

Error: In unknown:0: 'std::exception': boost::filesystem::directory_iterator::operator++: Transport endpoint is not connected: "/tmp/exe_cache/pyg-padding"
Error raised in:
  [0] popart::TrainingSession::createFromOnnxModel
  [1] Compiler::initSession
  [2] LowerToPopart::compile
  [3] compileWithManualTracing

4_small_graph_batching_with_packing.ipynb cell 20 poptorch_model.train() ... fails with

Error: In unknown:0: 'std::exception': boost::filesystem::directory_iterator::operator++: Transport endpoint is not connected: "/tmp/exe_cache/pyg-packing"
Error raised in:
  [0] popart::TrainingSession::createFromOnnxModel
  [1] Compiler::initSession
  [2] LowerToPopart::compile
  [3] compileWithManualTracing

The remainder of these 3 notebooks are blocked from running.

anjleeg-gcai commented 1 year ago

Hi, I’m unable to reproduce these errors. Can you give me more information on the runtime details? Which image is your container pointing to? Which machine type are you running on?