No module named 'omni.isaac.sensor'

[Pipe-Runner]

I am trying to use a Lidar sensor in my task. Oddly when I use the import in a small test code like the following, it works just fine.

import os
from omni.isaac.kit import SimulationApp

# Simple example showing how to start and stop the helper
simulation_app = SimulationApp({"headless": True})

### Perform any omniverse imports here after the helper loads ###
from omni.isaac.sensor import RotatingLidarPhysX

class LidarView(RotatingLidarPhysX):
    ...
    ...

simulation_app.update()  # Render a single frame
simulation_app.close()  # Cleanup application

I am using VecEnvBase to create and env and test out the task. No training is involved here. But here I am getting module not found error. I am using the same environment that I used to run the previous script by the way.

import torch
from ..tasks.franka_cabinet_lidar.config import TASK_CFG
from omni.isaac.gym.vec_env import VecEnvBase

TASK_CFG["headless"] = True
TASK_CFG["task"]["env"]["numEnvs"] = 1

env = VecEnvBase(headless=TASK_CFG["headless"])

from omniisaacgymenvs.utils.config_utils.sim_config import SimConfig  # noqa
from ..tasks.franka_cabinet_lidar.task import FrankaCabinetLidarTask  # noqa

sim_config = SimConfig(TASK_CFG)
task = FrankaCabinetLidarTask(name="Debug", sim_config=sim_config, env=env)
env.set_task(
    task=task,
    sim_params=sim_config.get_physics_params(),
    backend="torch",
    init_sim=True,
)

env._world.reset()
obs = env.reset()
while env._simulation_app.is_running():
    # action, _states = model.predict(obs)
    action = torch.zeros(
        (task._num_envs, task._num_actions),
        dtype=torch.float,
        device=task._device,
    )
    obs, rewards, dones, info = env.step(action)

env.close()

FrankaCabinetLidarTask has the from omni.isaac.sensor import RotatingLidarPhysX import inside it.

[Pipe-Runner]

Just to confirm that there is no error in my code, I just added the import in the cartpole example in OIGE and I get the same error. I have also tested with the python provided by Isaac and also python env with conda. Same error in both cases.

[kellyguo11]

Hi there, for performance reasons, we only include a small set of extensions that are required for the gym examples. For additional dependencies, such as omni.isaac.sensor, please add it to the app file omni.isaac.sim.python.gym.headless.kit under isaac_sim/apps, at the bottom of the file, or enable it from the Extensions Manager under Windows in the top menu bar. Alternatively, you can specify a different app file to use when defining VecEnvBase with argument experience=f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.kit', which should include the full set of extension dependencies.

[Pipe-Runner]

@kellyguo11 Thank you. I'll give this a try.

[Pipe-Runner]

@kellyguo11 That seems to have solved the problem but I am getting a new error now:

2023-10-31 18:06:12 [13,275ms] [Error] [carb.physx.python] TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

At:
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py(41): create_tensor_from_list
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py(181): _data_acquisition_callback
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(463): step
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/world/world.py(380): step
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py(218): reset
  /home/piperunner/Projects/research/project-mirage/src/mirage/eval/debug_eval.py(23): <module>
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/runpy.py(86): _run_code
  /home/piperunner/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/runpy.py(196): _run_module_as_main

I haven't tried reading any data with the sensor yet; I have only imported it.

This seems to have been reported before and someone has attempted to fix the problem by retrofitting the code in the _data_acquisition_callback function in exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py. Do you think it's a valid approach?

[chenyafei-code]

Hi there, for performance reasons, we only include a small set of extensions that are required for the gym examples. For additional dependencies, such as omni.isaac.sensor, please add it to the app file omni.isaac.sim.python.gym.headless.kit under isaac_sim/apps, at the bottom of the file, or enable it from the Extensions Manager under Windows in the top menu bar. Alternatively, you can specify a different app file to use when defining VecEnvBase with argument experience=f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.kit', which should include the full set of extension dependencies.

I want to know how to add 'omni.isaac.sensor' to the app file

[mccunnj]

Hi there, for performance reasons, we only include a small set of extensions that are required for the gym examples. For additional dependencies, such as omni.isaac.sensor, please add it to the app file omni.isaac.sim.python.gym.headless.kit under isaac_sim/apps, at the bottom of the file, or enable it from the Extensions Manager under Windows in the top menu bar. Alternatively, you can specify a different app file to use when defining VecEnvBase with argument experience=f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.kit', which should include the full set of extension dependencies.

I want to know how to add 'omni.isaac.sensor' to the app file

I'm on win10. For me it was to edit the .kit file mentioned above in C:\Users\myUserName\AppData\Local\ov\pkg\isaac_sim-2023.1.1\apps add "omni.isaac.sensor" = {} to the bottom of the file.

[RealmX1]

Hi there, for performance reasons, we only include a small set of extensions that are required for the gym examples. For additional dependencies, such as omni.isaac.sensor, please add it to the app file omni.isaac.sim.python.gym.headless.kit under isaac_sim/apps, at the bottom of the file, or enable it from the Extensions Manager under Windows in the top menu bar. Alternatively, you can specify a different app file to use when defining VecEnvBase with argument experience=f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.kit', which should include the full set of extension dependencies.

@kellyguo11 Please add this to documentation. Took me 3 hours trying to figure out the source of my problem before I arrive at this post. Thanks.