Substeps bug

[enaitzb]

If you are submitting a bug report, please fill in the following details and use the tag [bug].

Describe the bug

Error when training the Reach task with the Franka robot using the sim.substeps parameter with SB3. The error only occurs when trained in headless mode.

Steps to reproduce

def __post_init__(self):
        """Post initialization."""
        # general settings
        self.decimation = 2
        self.episode_length_s = 12.0
        self.viewer.eye = (3.5, 3.5, 3.5)
        # simulation settings
        self.sim.dt = 1/60
        self.sim.substeps = 20

   python source/standalone/workflows/sb3/train.py --task Isaac-Reach-Franka-v0 --num_envs 16 --headless

-->

System Info

Describe the characteristic of your environment:

• Commit: 8816fb7
• Isaac Sim Version: 4.0
• OS: Ubuntu 22.04.3 LTS
• GPU: NVIDIA GeForce RTX 4090
• CUDA: 12.2
• GPU Driver: 535.146.02

Checklist

• [x] I have checked that there is no similar issue in the repo (required)
• [] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

[Dhoeller19]

Hi @enaitzb, can you share your sb3 config? I cannot reproduce with rsl_rl

[enaitzb]

Hi @Dhoeller19, these is my sb3 config

# Reference: https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/ppo.yml#L32
seed: 42
policy: 'MlpPolicy'
n_timesteps: !!float 1e5
batch_size: 32
n_steps: 512
gamma: 0.9
learning_rate: 0.0003 #
ent_coef: 0.01 #
clip_range: 0.2  #
n_epochs: 4 #
gae_lambda: 0.98 #
max_grad_norm: 0.5 #
vf_coef: 0.5 #

I have tried again and the error does not occur when the dt is 1/60. However, I have tried with:

• self.sim.dt = 1/40
• self.sim.dt = 1/120
• self.sim.dt = 0.008

And the error keeps popping up.

It have too these error line that looks that the scene cannot be created

2024-06-11 07:37:41 [7,271ms] [Error] [omni.physx.tensors.plugin] Failed to create simulation view: no active physics scene found

I also tried the lift task with rsl_rl and got the same error.

    def __post_init__(self):
        """Post initialization."""
        # general settings
        self.decimation = 2
        self.episode_length_s = 5.0
        # simulation settings
        self.sim.dt = 0.01  # 100Hz

        self.sim.substeps = 20
        self.sim.physx.bounce_threshold_velocity = 0.2
        self.sim.physx.bounce_threshold_velocity = 0.01
        self.sim.physx.gpu_found_lost_aggregate_pairs_capacity = 1024 * 1024 * 4
        self.sim.physx.gpu_total_aggregate_pairs_capacity = 16 * 1024
        self.sim.physx.friction_correlation_distance = 0.00625

# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from omni.isaac.lab.utils import configclass

from omni.isaac.lab_tasks.utils.wrappers.rsl_rl import (
    RslRlOnPolicyRunnerCfg,
    RslRlPpoActorCriticCfg,
    RslRlPpoAlgorithmCfg,
)

@configclass
class LiftCubePPORunnerCfg(RslRlOnPolicyRunnerCfg):
    num_steps_per_env = 24
    max_iterations = 1500
    save_interval = 50
    experiment_name = "franka_lift"
    empirical_normalization = False
    policy = RslRlPpoActorCriticCfg(
        init_noise_std=1.0,
        actor_hidden_dims=[256, 128, 64],
        critic_hidden_dims=[256, 128, 64],
        activation="elu",
    )
    algorithm = RslRlPpoAlgorithmCfg(
        value_loss_coef=1.0,
        use_clipped_value_loss=True,
        clip_param=0.2,
        entropy_coef=0.006,
        num_learning_epochs=5,
        num_mini_batches=4,
        learning_rate=1.0e-4,
        schedule="adaptive",
        gamma=0.98,
        lam=0.95,
        desired_kl=0.01,
        max_grad_norm=1.0,
    )

[Dhoeller19]

@enaitzb I confirm I can reproduce and I also have a fix in the pipeline already.