[Question] fps drops significantly over time

[oxkitsune]

Question

I'm training a simple PPO agent in a mujoco gym env, however I've noticed that over the course of training the FPS goes down significantly. The fps keeps going down, even after the updates have started.

I'm wondering what could be the cause of this?

Additional context

• I'm training on gpu, although running on cpu yields similar results
• I'm training with the following parameters:
  • n_envs: 1
  • n_epochs: 10
  • n_steps: 1024
• I've read DLR-RM/stable-baselines3#1597, but I don't feel like any of the comments there apply to my case.
• Dependency versions:
  • Python 3.11
  • gymnasium 1.0 (master branch)
  • sbx 0.15
  • sb3 2.3.2

The following charts show cpu usage and fps over time, where this particular run took 17h on a machine with:

• cpu: 13th Gen Intel i9-13900KF (32) @ 6.000GHz
• gpu: NVIDIA GeForce RTX 4090

Checklist

• [x] I have read the documentation (required)
• [x] I have checked that there is no similar issue in the repo (required)

[araffin]

what mujoco version do you use? SB3 is only fully compatible with 0.29.1 for now.

Do you have the same issue with built-env mujoco env? (for instance HalfCheetah-v4)

I haven't experienced that so far, might do some runs that I will log in open rl benchmark later.