isaac-sim / IsaacLab

Unified framework for robot learning built on NVIDIA Isaac Sim
https://isaac-sim.github.io/IsaacLab
Other
2.03k stars 816 forks source link

Issue with FrameTransformer and two articulations #662

Open mikelasa opened 3 months ago

mikelasa commented 3 months ago

Hello, I started doing a simple version of a transformer from my robot link0 to the link7 following the run_frame_tutorial.py. I created a FrameTransformerCfg which takes the link0 as source and link 7 as target and tried to create the actual FrameTransformer with it (This transformer is located into my class TableTopSceneCfg(InteractiveSceneCfg) interactive scene):

#frame transformer
    ee_pad_frame_transformer_cfg = FrameTransformerCfg(
        #prim_path="/World/envs/env_0/Bin",
        #prim_path="{ENV_REGEX_NS}/Bin"
        prim_path="/World/envs/env_0/Robot1/panda_link0",
        debug_vis=False,
        target_frames=[
            FrameTransformerCfg.FrameCfg(prim_path="/World/envs/env_0/Robot1/panda_link7",
                                         name="pad_ee1_transform")
            #FrameTransformerCfg.FrameCfg(prim_path="/World/envs/env_0/Robot2/rectangular_pad",
                                         #name="pad_ee2_transform"),
        ],
        visualizer_cfg=VisualizationMarkersCfg(
            prim_path="/Visuals/ObjectFrameTransformer",
            markers={
                "frame_ee1_pad": sim_utils.UsdFileCfg(
                    usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/UIElements/frame_prim.usd",
                    scale=(0.1, 0.1, 0.1),
                )
            }
        )
    )
    ee_pad_frame_transformer = FrameTransformer(ee_pad_frame_transformer_cfg)

Commenting out ee_pad_frame_transformer = FrameTransformer(ee_pad_frame_transformer_cfg) I'm able to launch the simulation and see in Visuals the created "ObjectFrameTransformer" and also the visualization of the frame in the end effector (attached image). The moment I try to create the transformer sensor with this configuration I have the following error which I guess is related to isaac sim? (I dont understand the error very well):

Traceback (most recent call last):
  File "/home/mikel/IsaacOrbitLFD/IsaacLab/source/standalone/cartesian_impedance_control/env/bimanual_robot_pads.py", line 26, in <module>
    from bimanual_env_pad_cfg import BimanualLiftPressEnvCfg
  File "/home/mikel/IsaacOrbitLFD/IsaacLab/source/standalone/cartesian_impedance_control/env/bimanual_env_pad_cfg.py", line 399, in <module>
    class BimanualLiftPressEnvCfg(ManagerBasedEnvCfg):
  File "/home/mikel/IsaacOrbitLFD/IsaacLab/source/standalone/cartesian_impedance_control/env/bimanual_env_pad_cfg.py", line 403, in BimanualLiftPressEnvCfg
    scene: TableTopSceneCfg = TableTopSceneCfg(num_envs=2, env_spacing=2.5)
  File "<string>", line 15, in __init__
  File "/home/mikel/IsaacOrbitLFD/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/utils/configclass.py", line 418, in _wrap
    return deepcopy(f)
  File "/home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'carb.events._events.ISubscription' object
2024-07-09 14:48:15 [21,293ms] [Warning] [carb] [Plugin: omni.sensors.tiled.plugin] Module /home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/extscache/omni.sensors.tiled-0.0.3+106.0.0.lx64.r/bin/libomni.sensors.tiled.plugin.so remained loaded after unload request
2024-07-09 14:48:15 [21,298ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /home/mikel/.local/share/ov/pkg/isaac-sim-4.0.0/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
2024-07-09 14:48:15 [21,300ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,300ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,300ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2024-07-09 14:48:15 [21,301ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Core' for removal
2024-07-09 14:48:15 [21,468ms] [Warning] [carb] Recursive unloadAllPlugins() detected!

My goal is to use the FrameTransformer to compute the transformation from an object in the scene to both end effectors I have. I'm aware of the note telling that this function is meant to be used within an articulation. There is a possibility of achieving my goal using this class or should I do the computations by myself? (attached a sketch)

captura PXL_20240709_151113920

Mayankm96 commented 3 months ago

Where did you create the frame transformer sensor? Did you do it after the sim.reset() call?

mikelasa commented 3 months ago

Hi,

Is before env.reset() is called, I first create the ManagerBasedEnv and the call the reset. Also note that I have a contact sensor that is working well if it helps:

@configclass
class TableTopSceneCfg(InteractiveSceneCfg):
    """Configuration for a cart-pole scene."""

    # ground plane

    # lights

    # mount

    # Set Cube as object

    # contact sensor

    #frame transformer
    ee_pad_frame_transformer_cfg = FrameTransformerCfg(
        #prim_path="/World/envs/env_0/Bin",
        #prim_path="{ENV_REGEX_NS}/Bin"
        prim_path="/World/envs/env_0/Robot1/panda_link0",
        debug_vis=False,
        target_frames=[
            FrameTransformerCfg.FrameCfg(prim_path="/World/envs/env_0/Robot1/panda_link7",
                                         name="pad_ee1_transform")
            #FrameTransformerCfg.FrameCfg(prim_path="/World/envs/env_0/Robot2/rectangular_pad",
                                         #name="pad_ee2_transform"),
        ],
        visualizer_cfg=VisualizationMarkersCfg(
            prim_path="/Visuals/ObjectFrameTransformer",
            markers={
                "frame_ee1_pad": sim_utils.UsdFileCfg(
                    usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/UIElements/frame_prim.usd",
                    scale=(0.1, 0.1, 0.1),
                )
            }
        )
    )
    ee_pad_frame_transformer = FrameTransformer(ee_pad_frame_transformer_cfg)

Then goes the env:

@configclass
class BimanualLiftPressEnvCfg(ManagerBasedEnvCfg):

    """Configuration for the lifting environment."""
    # Scene settings
    scene: TableTopSceneCfg = TableTopSceneCfg(num_envs=2, env_spacing=2.5)
    # Basic settings
    observations: ObservationsCfg = ObservationsCfg()
    actions: ActionsCfg = ActionsCfg()
    events: EventCfg = EventCfg()

    def __post_init__(self):
        """Post initialization."""
        # camera settings 
        self.viewer.eye = [2.86341, 2.03233, 1.67289]

        # general settings
        self.decimation = 4
        self.episode_length_s = 5.0
        # simulation settings
        self.sim.dt = 0.001  # 1KHz

Finally the main:

def main():
    # parse the arguments
    env_cfg = BimanualLiftPressEnvCfg()
    env_cfg.scene.num_envs = args_cli.num_envs
    # setup base environment
    env = ManagerBasedEnv(cfg=env_cfg)
    obs, _ = env.reset()