search

DLR-RM / stable-baselines3

PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.
https://stable-baselines3.readthedocs.io
MIT License
9.18k stars 1.71k forks source link

Tensorboard log not updating with stable-baselines3 (SAC) #1419

Closed schen993 closed 1 year ago

schen993 commented 1 year ago

I am using the following libraries: stable-balines3 0.11.1 tensorboard 2.5.0 tensorflow 2.11.1

When I try to run the following code: tensorboard_log_path='./baselines/logs boundary = Monitor(env) callback = CustomCallback(eval_env=boundary,n_eval_episodes=1,eval_freq=1000) model = SAC(policy='MlpPolicy',env=boundary,tensorboard_log=tensorboard_log_path) model.learn(total_timesteps=1000000, callback=callback)

CustomCallback inherits from stable_baselines3.common.callbacks.EventCallback and has the following logging code in the _on_step() function: self.logger.record("eval/mean_reward",float(mean_reward) self.logger.record("eval/mean_ep_length",mean_ep_length)

And when I step through, the code in CustomCallback is able to be reached, with valid values for mean_reward and mean_ep_length, and the logger has been instantiated.

My environment inherits from gym.Env.

When I run the code above, the tensorboard log is created, but the file is 1kb, and the tensorboard doesn't show any data after 30000 steps.

When I run the same code with A2C, the log is created, updated, and shows on tensorboard.

I have also tried to wrap my environment with DummyVecEnv([lambda: env]), but the same thing happens - tensorboard shows data for A2C but is empty for SAC.

Please let me know if you need additional information. Thank you very much.

araffin commented 1 year ago

Hello, please use the issue template. It also seems you should upgrade your SB3 version.