Open 9B8DY6 opened 2 years ago
@9B8DY6 I solved it by downgrading the setuptools to version 59 as suggested here
pip install setuptools==59.5.0
@9B8DY6 I solved it by downgrading the setuptools to version 59 as suggested here
pip install setuptools==59.5.0
Thank you for your reply. @digvijayad
@9B8DY6 have you managed to run the model? I'm getting another error: version 'GLIBCXX_3.4.XX' not found
@9B8DY6 have you managed to run the model? I'm getting another error:
version 'GLIBCXX_3.4.XX' not found
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=1, worker_count=4, timeout=0:30:00) ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 567) of binary: /opt/conda/bin/python ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish /opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:70
No, I still got the problem of torch.distributed
That looks like you have a timeout of 30 minutes for your worker. Therefore, it timed out before it could finish. Not to sure about it. Have you finished building unprojections or are you just testing with default cityscapes ?
@9B8DY6 have you managed to run the model? I'm getting another error:
version 'GLIBCXX_3.4.XX' not found
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=1, worker_count=4, timeout=0:30:00) ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 567) of binary: /opt/conda/bin/python ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish /opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:70
No, I still got the problem of torch.distributed
Hi @9B8DY6,I am having this same issue. Did you manage to fix it?
I got error of disutils that has no 'version' and raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: I tried to install imaginaire with conda but it does not work. Would you tell me how to fix it? If I solve the problem of disutils with pip install setuptools, then may torch.distributed problem be also solved?