ValueError: Attempting to unscale FP16 gradients.

[Kyfafyd]

Get an error during stage 2.

Traceback (most recent call last):
  File "/mnt/petrelfs/wangzhao/HumanT2V/AnimateAnyone/train_svd.py", line 803, in <module>
    main(config)
  File "/mnt/petrelfs/wangzhao/HumanT2V/AnimateAnyone/train_svd.py", line 672, in main
    accelerator.clip_grad_norm_(
  File "/mnt/petrelfs/wangzhao/anaconda3/envs/consisti2v/lib/python3.10/site-packages/accelerate/accelerator.py", line 1925, in clip_grad_norm_
    self.unscale_gradients()
  File "/mnt/petrelfs/wangzhao/anaconda3/envs/consisti2v/lib/python3.10/site-packages/accelerate/accelerator.py", line 1888, in unscale_gradients
    self.scaler.unscale_(opt)
  File "/mnt/petrelfs/wangzhao/anaconda3/envs/consisti2v/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 284, in unscale_
    optimizer_state["found_inf_per_device"] = self._unscale_grads_(optimizer, inv_scale, found_inf, False)
  File "/mnt/petrelfs/wangzhao/anaconda3/envs/consisti2v/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 212, in _unscale_grads_
    raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.

[WJohnnyW]

same error, any solution?

[aleeyang]

same error