search

pytorch / elastic

PyTorch elastic training
BSD 3-Clause "New" or "Revised" License
730 stars 98 forks source link

EtcdStore: AttributeError: can't set attribute #152

Open vv-p opened 3 years ago

vv-p commented 3 years ago

Hi,

I have the following error when I try to run my code with torchelastic:

Creating EtcdStore as the c10d::Store implementation

Traceback (most recent call last):
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/distributed/launch.py", line 531, in main
 run_result = elastic_agent.run(spec.role)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/metrics/api.py", line 126, in wrapper
 result = f(*args, **kwargs)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/agent/server/api.py", line 680, in run
 result = self._invoke_run(role)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/agent/server/api.py", line 802, in _invoke_run
 self._initialize_workers(self._worker_group)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/metrics/api.py", line 126, in wrapper
 result = f(*args, **kwargs)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/agent/server/api.py", line 654, in _initialize_workers
 self._rendezvous(worker_group)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/metrics/api.py", line 126, in wrapper
 result = f(*args, **kwargs)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/agent/server/api.py", line 518, in _rendezvous
 store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous()
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/rendezvous/etcd_rendezvous.py", line 157, in next_rendezvous
 store = self._rdzv_impl.setup_kv_store(rdzv_version)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/rendezvous/etcd_rendezvous.py", line 975, in setup_kv_store
 return EtcdStore(etcd_client=self.client, etcd_store_prefix=store_path)
 File "/opt/conda/lib/python3.8/site-packages/torchelastic/rendezvous/etcd_rendezvous.py", line 997, in __init__
 self.timeout = (
AttributeError: can't set attribute

Steps to reproduce:

>>> from torch.distributed import Store
>>> class A(Store):
...     def __init__(self):
...             super().__init__()
...             self.timeout = 1
...
>>> a = A()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in __init__
AttributeError: can't set attribute

I've tried several different versions of torch and torchelastic (latest stable included) but nothing happened, error is still here. Can you help me please, what does this error mean ? How I can fix it ?

os centos 7 python python3.8.3 torch 1.9.0 torchelastic 0.2.2 python-etcd 0.4.5

melgor commented 3 years ago

I also faced this problem, but I managed to resolve it. In my case, it was just using torchelastic instead of torch for starting the jobs as torchelastic is now deprecated (https://pytorch.org/blog/pytorch-1.9-released/#beta-torchelastic-is-now-part-of-core)

So exactly, I changed: python3 -m torchelastic.distributed.launch to (documentation) python3 -m torch.distributed.run

And then, everything just started to work again.