Closed anirudhs001 closed 9 months ago
Hello, does that happens without gSDE? (and with ReLU instead of Tanh)
heyy, nope, it doesn't happen without gSDE (or hasn't happened so far atleast. Current run has done 20k updates over 60 million timesteps). I haven't tried using ReLU with gSDE yet.
Such a problem also occurred when I used ppo, but I thought it was a mistake that occurred when I modified other parts. Now I am not sure.
use_expln=True
should solve the problem, see https://github.com/DLR-RM/rl-baselines3-zoo/issues/427#issuecomment-1835541907
I had this same error, but in my case was caused by mixed precision, running the training within with th.xpu.amp.autocast(enabled=True, dtype=th.bfloat16). In the AdamW
optimizer step, the first param found in self.policy.optimizer.param_groups[0]['params'][0]
, (which I think is loss
, but I'm not sure) was being set to nan
for some reason. This happened somtimes after many hours of training, apparently when the loss value was getting high (around -40).
edit:
After removing the autocast I got the same error again, just 40 minutes of training later.
I have use_sde=False
, unlike in the issue. Shall I start another issue?
Traceback (most recent call last):
File "/home/arek/projects/arek-chess/arek_chess/training/run.py", line 267, in <module>
path = LOG_PATH if not args.env else os.path.join(LOG_PATH, args.env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arek/projects/arek-chess/arek_chess/training/run.py", line 147, in train
model.learn(total_timesteps=TOTAL_TIMESTEPS, reset_num_timesteps=reset_num_timesteps, tb_log_name=env_name, callback=TensorboardActionHistogramCallback()) # progress_bar=True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/stable_baselines3/ppo/ppo.py", line 316, in learn
return super().learn(
^^^^^^^^^^^^^^
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 309, in learn
self.train()
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/stable_baselines3/ppo/ppo.py", line 217, in train
values, log_prob, entropy = self.policy.evaluate_actions(rollout_data.observations, actions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/stable_baselines3/common/policies.py", line 745, in evaluate_actions
distribution = self._get_action_dist_from_latent(latent_pi)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/stable_baselines3/common/policies.py", line 701, in _get_action_dist_from_latent
return self.action_dist.proba_distribution(mean_actions, self.log_std)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/stable_baselines3/common/distributions.py", line 164, in proba_distribution
self.distribution = Normal(mean_actions, action_std)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/torch/distributions/normal.py", line 56, in __init__
super().__init__(batch_shape, validate_args=validate_args)
File "/home/arek/.pyenv/versions/env/lib/python3.11/site-packages/torch/distributions/distribution.py", line 68, in __init__
raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (261760, 1)) of distribution Normal(loc: torch.Size([261760, 1]), scale: torch.Size([261760, 1])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan],
[nan],
[nan],
...,
[nan],
[nan],
[nan]], device='xpu:0', grad_fn=<AddmmBackward0>)
🐛 Bug
the
log_std
tensor gets filled completely with NaNs and causes aValueError
exception during training with PPO. have tried using bothuse_expln=True/False
as mentioned in https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/issues/146The tensorboard graphs just before the exception:![image](https://github.com/DLR-RM/stable-baselines3/assets/52282675/6123979c-763e-4fdc-9fc9-375af1198b04)
Code example
Relevant log output / Error message
System Info
installed via pip output of python -c 'import stable_baselines3 as sb3; sb3.get_system_info()' :
Checklist