Invalid probability value in tensor when running mpo

[kan-s0]

Describe the bug RuntimeError when running mpo

To Reproduce

python main.py --config.mpo.atari --env.name breakout --sync

When config is modified with the values shown in the paper, it occurs faster and more frequently.

Expected behavior

• An error occurred when calculating multinomial method with pi from Actor network.
• RuntimeError: probability tensor contains either inf, nan or element < 0

Screenshots

training graph

• default config, green, also causes an error at 7M.

error txt

mpo generated agent code

Development Env. (OS, version, libraries):

• linux
• V4XLARGE
• python 3.7.11
• jorldy:0.3.0

Additional context

• Even with default config, an error sometimes occurs after a lot of learning.
• If you set the config to the value shown in the paper, you get a much higher score at the beginning, but an error quickly occurs.