Closed jkuball closed 5 years ago
Hello,
I installed via pip install -e .
What is your version? master? (v2.5.1?)
Could you provide the hyperparameters used, and did you see any other metric having too high values using tensorboard ? Did you have the same problem with other algorithms? (A2C for instance)
Unfortunately, this is surely due to either your environment or the chosen hyper-parameters.
Hi! Thanks for the fast answer.
What is your version? master? (v2.5.1?)
I was on bea2eed5b45cce875a762c55f342b1ba4e08fd3a and have now updated to the latest version.
Did you have the same problem with other algorithms? (A2C for instance)
I did not test this. I can, but I really need to use PPO2.
Could you provide the hyperparameters used [...]
Mostly I'm using the default values, but chose the batches relatively arbitrarily. I have a vecenv with 256 agents, a trajectory horizon of 128 and a nminibatch of 2048.
[...] and did you see any other metric having too high values using tensorboard ?
I don't think so, unfortunately I don't have the data right now to review. I'm currently generating a new set.
Unfortunately, this is surely due to either your environment or the chosen hyper-parameters.
That might very well the case, I'm really unexperienced in the field of reinforcement learning. There is a good chance I chose bad hyperparameters. (It does help to hear that you think that this is the problem and nothing in the PPO implementation itself.)
My environment is a very sparse rewarding one, so I wrote a vecenv curiosity wrapper using Keras. Also I'm using the VecNormalize
wrapper of this project to normalize the rewards.
And now that I'm writing this, I have a suspicion. Since I'm normalizing the sum of the intrinsic and extrinsic rewards -- and the extrinsic reward is mostly 0 -- could it be that it does explode when the environment is "solved", because then the extrinsic reward is just too high? I set this to 100 which is probably really far from the moving average in the VecNormalize wrapper. Could it be that a reward that is too high causes this issue?
but I really need to use PPO2.
I don't see obvious reason not to test A2C (which is really close to ppo) or another algorithm (depending on what action space you are using), except if you have done custom changes to it.
Could it be that a reward that is too high causes this issue?
It is hard to tell without looking at log of data and knowing the problem. But this more technical support and we don't do that, mainly because we want to focus on stable-baselines (and we already have only little time for doing so), not a particular custom environment.
So, I would suggest you to do more investigation, to track down where does the NaN comes from, and if you find clues that it may come from the algorithm, then please come back to us so we can fix that ;)
I don't see obvious reason not to test A2C (which is really close to ppo) or another algorithm
You're probably right. There are some reasons in my case, but that's another story.
But this more technical support and we don't do that [...]
Yeah, I wasn't sure whether this really was a bug or not, but it looks like it's an oversight on my side. That's good to know, now I have to debug more and more. Thank you for your time!
For everyone that stumbles upon this issue via google: For my case it looks like I had an entropy coefficient that was way too high.
The fact that bad chosen hyperparameters can result in NaNs inside the gradients calculation threw me really off, I'm closing this now! Thanks for the pointer!
Good to know, what was the magnitude of the entropy coefficient? (usually it is around 0.01 or smaller for PPO/A2C)
In the way of aimlessly testing parameters it ended up set to 0.5 which is arguably way too high. I used this blogpost to familiarize myself with the usual parameter ranges now. Maybe it's good to add something like "usually between x and y" to the documentation for all parameters? (On the other hand, this is not the duty of stable-baselines, but it might be helpful for beginners.)
Maybe it's good to add something like "usually between x and y" to the documentation for all parameters?
I would rather recommend looking at:
rather than having a pre-defined range.
Also, you should use at first automatic parameter tuning (available in the rl zoo) which saves a lot of effort compared to tuning by hand ;).
this is not the duty of stable-baselines
I agree that this is not the duty of SB. And if you change the default hyperparams, you should know what you are doing.
First of all, I'm not really sure whether this is a problem on my side or a bug on your side. But I'm trying to debug this for some days now and I really don't know what to do anymore.
Bug description
The bug I'm facing is easily described: while training I get NaN values while training a MlpPolicy with PPO2 on a custom environment I'm writing for my master's thesis.
The stacktrace is the following:
It looks like the NaNs are occuring in this call of
tf.gradients
. For further debugging I added some assertions:With those assertions added, I'm really sure that the
tf.gradients
call is the problem and the NaNs aren't propagated from theloss
variable, since the gradient with the index of 14 is the one that raises the error.Googling leads me to the assumption that this has to do with the numerical instability of the gradient calculation, so I thought it might be possible to add an epsilon ontop of the
loss
variable.Sadly, this doesn't help and the error persists. I'm not really sure what to do next and it doesn't help that any test needs multiple hours to verify.
Code example
I can't provide a minimal code example and the problem occurs only after one to three hour training on my machine, but I'll happily test anything anyone suggests. I am grateful for every comment, I really have to fix this.
System Info
I don't think this is a hardware- or installation problem, but I'll add the system info:
pip install -e .