Open ULudo opened 1 year ago
@ULudo do you get the same issue if you enable the following flags:
config.rl_module(_enable_rl_module_api=True)
config.training(_enable_learner_api=True)
We probably won't be merging any fixes to our old impala implementation that is built on our old training stack, but the above flags enable our new training stack.
The original error is coming because for some reason NANs are being introduced into the policy gradient path, and this causes an error to occur during training. Training is occuring on a python thread so the base error will be something like "the learner thread died while training" since there was an error on training code on the thread.
We don't have this setup with our new training stack, and we use a whole different set of models. We added in some tricks for stability to the RLModules used for IMPALA/APPO, so give that a try first and then let me know if this helps.
Thank you for your reply, it took me some time to test this. Using these two configuration flags helps, but the error still occurs, just at a later training point.
Script:
from ray.rllib.algorithms.impala import ImpalaConfig
from ray.rllib.algorithms import Algorithm
import gymnasium as gym
env_name = "MountainCarContinuous-v0"
config = ImpalaConfig()
config = config.training(gamma=0.99, lr=0.0003, train_batch_size=256, _enable_learner_api=True)
config = config.framework(framework="torch")
config = config.resources(num_gpus=1)
config = config.rollouts(num_rollout_workers=1, num_envs_per_worker=1, rollout_fragment_length=256)
config = config.environment(env=env_name)
config = config.rl_module(_enable_rl_module_api=True)
algo:Algorithm = config.build()
eval_returns = []
for i in range(1000):
print(f"Training iteration {i}")
result = algo.train()
Output:
python.exe : 2023-06-28 07:20:41,522 WARNING algorithm_config.py:2451 -- Setting `exploration_config={}` because you
set `_enable_rl_modules=True`. When RLModule API are enabled, exploration_config can not be set. If you want to
implement custom exploration behaviour, please modify the `forward_exploration` method of the RLModule at hand. On
configs that have a default exploration config, this must be done with `config.exploration_config={}`.
At line:1 char:1
+ python.exe .\src\impala_mountain_car.py *>&1 > .\train_mc_res.txt
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (2023-06-28 07:2...ion_config={}`.:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\algorithms\algorithm.py:442: RayDeprecationWarning:
This API is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env
variable PYTHONWARNINGS="ignore::DeprecationWarning"
`UnifiedLogger` will be removed in Ray 2.7.
return UnifiedLogger(config, logdir, loggers=None)
C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\tune\logger\unified.py:53: RayDeprecationWarning: This API
is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable
PYTHONWARNINGS="ignore::DeprecationWarning"
The `JsonLogger interface is deprecated in favor of the `ray.tune.json.JsonLoggerCallback` interface and will be
removed in Ray 2.7.
self._loggers.append(cls(self.config, self.logdir, self.trial))
C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\tune\logger\unified.py:53: RayDeprecationWarning: This API
is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable
PYTHONWARNINGS="ignore::DeprecationWarning"
The `CSVLogger interface is deprecated in favor of the `ray.tune.csv.CSVLoggerCallback` interface and will be removed
in Ray 2.7.
self._loggers.append(cls(self.config, self.logdir, self.trial))
C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\tune\logger\unified.py:53: RayDeprecationWarning: This API
is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable
PYTHONWARNINGS="ignore::DeprecationWarning"
The `TBXLogger interface is deprecated in favor of the `ray.tune.tensorboardx.TBXLoggerCallback` interface and will be
removed in Ray 2.7.
self._loggers.append(cls(self.config, self.logdir, self.trial))
2023-06-28 07:20:44,034 INFO worker.py:1636 -- Started a local Ray instance.
[2m[36m(RolloutWorker pid=8208)[0m
C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\gymnasium\utils\passive_env_checker.py:233:
DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`. (Deprecated NumPy 1.24)
[2m[36m(RolloutWorker pid=8208)[0m if not isinstance(terminated, (bool, np.bool8)):
2023-06-28 07:20:52,571 INFO trainable.py:173 -- Trainable.setup took 11.018 seconds. If your trainable is slow to
initialize, consider setting reuse_actors=True to reduce actor creation overheads.
2023-06-28 07:20:52,571 WARNING util.py:68 -- Install gputil for GPU system monitoring.
Training iteration 0
Training iteration 1
...
Training iteration 928
Training iteration 929
Traceback (most recent call last):
File ".\src\impala_mountain_car.py", line 23, in <module>
result = algo.train()
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\tune\trainable\trainable.py", line 389, in train
raise skipped from exception_cause(skipped)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\tune\trainable\trainable.py", line 386, in train
result = self.step()
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\algorithms\algorithm.py", line 803, in step
results, train_iter_ctx = self._run_one_training_iteration()
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\algorithms\algorithm.py", line 2853, in
_run_one_training_iteration
results = self.training_step()
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\algorithms\impala\impala.py", line 720, in
training_step
train_results = self.learn_on_processed_samples()
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\algorithms\impala\impala.py", line 960, in
learn_on_processed_samples
result = self.learner_group.update(
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\core\learner\learner_group.py", line 198, in
update
self._learner.update(
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\core\learner\learner.py", line 864, in update
result = self._update(minibatch)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\core\learner\learner.py", line 1095, in
_update
loss = self.compute_loss(fwd_out=fwd_out, batch=tensorbatch)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\core\learner\learner.py", line 677, in
compute_loss
module_results = self.compute_loss_per_module(
File
"C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\algorithms\impala\torch\impala_torch_learner.py",
line 29, in compute_loss_per_module
target_policy_dist = action_dist_class_train.from_logits(
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\models\torch\torch_distributions.py", line
217, in from_logits
return TorchDiagGaussian(loc=loc, scale=scale)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\models\torch\torch_distributions.py", line
189, in __init__
super().__init__(loc=loc, scale=scale)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\models\torch\torch_distributions.py", line
27, in __init__
self._dist = self._get_torch_distribution(*args, **kwargs)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\ray\rllib\models\torch\torch_distributions.py", line
192, in _get_torch_distribution
return torch.distributions.normal.Normal(loc, scale)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\torch\distributions\normal.py", line 56, in __init__
super().__init__(batch_shape, validate_args=validate_args)
File "C:\Users\ULudo\miniconda3\envs\test_env\lib\site-packages\torch\distributions\distribution.py", line 62, in
__init__
raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (256, 1)) of distribution Normal(loc: torch.Size([256, 1]), scale:
torch.Size([256, 1])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan],
[nan]], grad_fn=<SplitBackward0>)
I have also encountered this problem. Have you resolved it?
python.exe : 2023-06-28 07:20:41,522 WARNING algorithm_config.py:2451 -- Setting exploration_config={}
because you
set _enable_rl_modules=True
. When RLModule API are enabled, exploration_config can not be set. If you want to
implement custom exploration behaviour, please modify the forward_exploration
method of the RLModule at hand. On
configs that have a default exploration config, this must be done with config.exploration_config={}
.
Having this problem as well on RLLib 2.7.1. After a few training iterations gradients go to nan:
(APPO pid=2911932) ValueError: Expected parameter loc (Tensor of shape (500, 10)) of distribution Normal(loc: torch.Size([500, 10]), scale: torch.Size([500, 10])) to satisfy the constraint Real(), but found invalid values:
(APPO pid=2911932) tensor([[nan, nan, nan, ..., nan, nan, nan],
(APPO pid=2911932) [nan, nan, nan, ..., nan, nan, nan],
(APPO pid=2911932) [nan, nan, nan, ..., nan, nan, nan],
(APPO pid=2911932) ...,
(APPO pid=2911932) [nan, nan, nan, ..., nan, nan, nan],
(APPO pid=2911932) [nan, nan, nan, ..., nan, nan, nan],
(APPO pid=2911932) [nan, nan, nan, ..., nan, nan, nan]], device='cuda:0',
(APPO pid=2911932) grad_fn=<SplitBackward0>)
What happened + What you expected to happen
Problem When using the IMPALA algorithm in RLlib the training process crashes after a while with the error message „RuntimeError: The learner thread died while training!“. This already happens when using a single worker. I get this error when using a custom environment, but this also occurs when using benchmark environments.
Full error message
Versions / Dependencies
Setup
Reproduction script
Environment setup
conda create -n test_env python=3.8 ipython
conda activate test_env
pip install "ray[rllib]"
pip install absl-py>=0.6.1
pip install gin-config==0.1.3
pip install tensorflow-probability>=0.9.0
pip install tensorflow
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
pip install chardet
Reproduction script
Additional Information Using the following additional settings delays the occurrence of the error a little bit:
Issue Severity
High: It blocks me from completing my task.