search

OFA-Sys / ONE-PEACE

A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities
Apache License 2.0
935 stars 57 forks source link

AttributeErrorAttributeError: : 'Adam' object has no attribute 'unscale_grads''Adam' object has no attribute 'unscale_grads' #33

Closed ZhanRao closed 11 months ago

ZhanRao commented 11 months ago

File "/home/ubuntu/hao/live_view2/ONE_PEACE/fairseq/fairseq/distributed/utils.py", line 330, in distributed_main return func(*args, *kwds) File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/trainer.py", line 841, in train_step return func(args, kwds) File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/trainer.py", line 841, in train_step main(cfg, kwargs) File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/run_scripts/visual_grounding/../../train.py", line 177, in main valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
self.task.optimizer_step( File "/home/ubuntu/anaconda3/envs/torch/lib/python3.9/contextlib.py", line 79, in inner File "/home/ubuntu/hao/live_view2/ONE_PEACE/fairseq/fairseq/tasks/fairseq_task.py", line 526, in optimizer_step self.task.optimizer_step( self.task.optimizer_step( File "/home/ubuntu/hao/live_view2/ONE_PEACE/fairseq/fairseq/tasks/fairseq_task.py", line 526, in optimizer_step

return func(*args, **kwds)  File "/home/ubuntu/hao/live_view2/ONE_PEACE/fairseq/fairseq/tasks/fairseq_task.py", line 526, in optimizer_step

File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/run_scripts/visual_grounding/../../train.py", line 272, in train optimizer.step() File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 126, in step log_output = trainer.train_step(samples, empty_cache=(i == 0)) File "/home/ubuntu/anaconda3/envs/torch/lib/python3.9/contextlib.py", line 79, in inner optimizer.step() File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 126, in step optimizer.step() File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 126, in step self._unscale_grads()return func(*args, **kwds)

File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 87, in _unscale_grads File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/trainer.py", line 841, in train_step self._unscale_grads()
self._unscale_grads() File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 87, in _unscale_grads

  File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 87, in _unscale_grads

self.wrapped_optimizer.optimizer.unscale_grads(self._multiply_factor) AttributeError : self.wrapped_optimizer.optimizer.unscale_grads(self._multiply_factor)'Adam' object has no attribute 'unscale_grads'

self.wrapped_optimizer.optimizer.unscale_grads(self._multiply_factor)

AttributeErrorAttributeError: : 'Adam' object has no attribute 'unscale_grads''Adam' object has no attribute 'unscale_grads'

self.task.optimizer_step(

File "/home/ubuntu/hao/live_view2/ONE_PEACE/fairseq/fairseq/tasks/fairseq_task.py", line 526, in optimizer_step optimizer.step() File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 126, in step self._unscale_grads() File "/home/ubuntu/hao/live_view2/ONE_PEACE/one_peace/optim/fp16_optimizer_memory_efficent.py", line 87, in _unscale_grads self.wrapped_optimizer.optimizer.unscale_grads(self._multiply_factor) AttributeError: 'Adam' object has no attribute 'unscale_grads'

logicwong commented 11 months ago

Thanks for the feedback, we have fixed the bug. Please check out our latest commit and give it a try. Additionally, we recommend installing Apex to save GPU memory usage.

git clone https://github.com/NVIDIA/apex
cd apex && pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./
ZhanRao commented 11 months ago

Thank you very much for your help, the problem has been solved