Clone Envs Multiple Times to Implement Multi-task Learning Environments

[AlanSunHR]

Hi,

I am implementing a multi-task learning environment, where I have two (or possibly more) tasks. each consisting of a robot and an object. Say if the total number of the environments is 8, each task will have 4 environments.

I want to clone the two task environments so that 0-3 envs learn the task1 and 4-8 envs learn task2. I firstly put the source usd of task 1 to env0, and task 2 to env4. Then I use the cloner to clone env0 to env0-3, and clone env4 to env4-8.

Everything looks fine but I found two problems:

1. The simulation becomes much slower (half of the speed than learning a single task just like other tasks in this repo)
2. The dynamics look pretty different than learning each task separatedly. The joint acceleration is almost doubled and the robots are shaking with 0.0 actions. (The exact same robot in a single-task setting does not shake at all)

Some additional information:

1. The robot in each task is the same, but in task1 it is fixed, and in task2 it is free-floating. I don't know if we can add a fixed joint before the simulation runs so I just use separated usds for the two tasks.
2. I set replicate_physics to False since setting it to True will trigger an exception.
3. I tried both 2022.2.1 and 2023.1.1 but problems are still there.

The following codes are the modified part of rl_task.py (inside function set_up_scene):

        source_path_list = []
        destination_paths_list = [] 
        for task in self.task_list:
            source_path = "/World/envs/env_{}".format(task.env_id_range[0]) # The min id of the task will be the source
            destination_paths = ["/World/envs/env_{}".format(i) for i in range(task.env_id_range[0], task.env_id_range[1]+1)]
            source_path_list.append(source_path)
            destination_paths_list.append(destination_paths)

        super().set_up_scene(scene)

        if self._task_cfg["sim"].get("add_ground_plane", True):
            self._ground_plane_path = "/World/defaultGroundPlane"
            collision_filter_global_paths.append(self._ground_plane_path)
            scene.add_default_ground_plane(prim_path=self._ground_plane_path,
                                           z_position=ground_z_position)

        for i, task in enumerate(self.task_list):
            source_path = source_path_list[i]
            destination_paths = destination_paths_list[i]

            # For each task, it will be located in the height specified by the variable self._task_spacing
            task_height = i*self._task_spacing
            env_positions = np.zeros((task.env_id_range[1]-task.env_id_range[0]+1, 3))
            env_positions[:, 2] = task_height
            _env_pos = self._cloner.clone(source_prim_path=source_path,
                                          root_path="/World/envs/env", 
                                          prim_paths=destination_paths, 
                                          position_offsets=env_positions,
                                          replicate_physics=False) # Always set replicate_physics to False
            _env_pos = torch.tensor(np.array(_env_pos), device=self._device, dtype=torch.float)
            # Set task env_pos
            if not hasattr(task, "_env_pos"):
                setattr(task, 
                        "_env_pos",
                        _env_pos)
            else:
                task._env_pos = _env_pos

            # Reset env positions so that the cloner can deal with the next cloning operation
            self._cloner._positions = None

        self._cloner.filter_collisions(
            self._env._world.get_physics_context().prim_path, "/World/collisions", sum(destination_paths_list, []), collision_filter_global_paths)

Do you have any ideas to solve the issue? Thanks!