search

robodhruv / visualnav-transformer

Official code and checkpoint release for mobile robot foundation models: GNM, ViNT, and NoMaD.
http://general-navigation-models.github.io
MIT License
425 stars 56 forks source link

Inquiry on Environment and Simulator Usage #14

Closed chungcode0218 closed 5 months ago

chungcode0218 commented 5 months ago

Dear author or experienced users, may I ask what environment and simulator you are using (for example, versions of Carla, Ubuntu, etc.)? Additionally, is there any ViNT-related discussion group where we can discuss together?

Dashora7 commented 5 months ago

We ended up using CARLA 0.9.14 and Ubuntu 22.04.2, although I assume there's some flexibility there. We modified the OATomobile wrapper which is an open-source gym API on CARLA. There isn't a discussion group, so feel free to raise issues when needed.

942411526 commented 3 months ago

raise issues when needed.

I test OATomobile in CARLA 0.9.10 but Hello from the pygame community. https://www.pygame.org/contribute.html 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=118144 LargeMemoryPoolOffset=249280 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped) 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=118144 LargeMemoryPoolOffset=249280 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped) 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=118144 LargeMemoryPoolOffset=249280 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped) 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=118144 LargeMemoryPoolOffset=249280 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped) 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=118144 LargeMemoryPoolOffset=249280 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped)

Dashora7 commented 3 months ago

This could be a number of things. Usually, these segfaults are due to some rendering issue regarding Vulkan or OpenGL, or maybe offscreen/on-screen things with memory capacity. I'd advise you to look at the file in OATomobile that actually runs CARLA. Specifically, look at the setup() function in oatomobile.utils.carla

By default it runs : [ os.path.join(os.environ.get("CARLA_ROOT"), "CarlaUE4.sh"), "-carla-rpc-port={}".format(port), '-opengl', "-quality-level=Epic" ]

See how you can tune this to work on the command line, and once it's working, edit the setup(). Maybe try something like adding the argument "-RenderOffScreen" to that list above, or reducing the quality level for lower memory usage.

942411526 commented 3 months ago

[ os.path.join(os.environ.get("CARLA_ROOT"), "CarlaUE4.sh"), "-carla-rpc-port={}".format(port), '-opengl', "-quality-level=Epic" ]

Thanks for your response I change to [ os.path.join(os.environ.get("CARLA_ROOT"), "CarlaUE4.sh"), "-carla-rpc-port={}".format(port), '-opengl', "-quality-level=Epic" ] and run the test code
import oatomobile

Initializes a CARLA environment.

environment = oatomobile.envs.CARLAEnv(town="Town01")

Makes an initial observation.

observation = environment.reset() done = False

while not done:

Selects a random action.

action = environment.action_space.sample() observation, reward, done, info = environment.step(action)

Renders interactive display.

environment.render(mode="human")

Book-keeping: closes

environment.close() But there are some problems
File "oatomobile_test.py", line 7, in observation = environment.reset() File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/core/rl.py", line 67, in reset return self.simulator.reset(*args, **kwargs) File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/simulators/carla/simulator.py", line 1725, in reset vehicle_id="vehicle.ford.mustang", File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/util/carla.py", line 265, in spawn_hero hero_bp = bl.find(vehicle_id) IndexError: blueprint 'vehicle.ford.mustang' not found

Dashora7 commented 3 months ago

Okay, so RenderingOffScreen fixed it? If so, and this is a new, separate issue, then maybe consult with this comment: https://github.com/carla-simulator/carla/issues/2590#issuecomment-950066180 . This might help you find what blueprint items there are for you to access. Are you sure your Content folder in CARLA is full with all blueprints? Can you bring in as many assets as you can?

942411526 commented 3 months ago

Okay, so RenderingOffScreen fixed it? If so, and this is a new, separate issue, then maybe consult with this comment: carla-simulator/carla#2590 (comment) . This might help you find what blueprint items there are for you to access. Are you sure your Content folder in CARLA is full with all blueprints? Can you bring in as many assets as you can?

there are some problems still pygame 1.9.6 Hello from the pygame community. https://www.pygame.org/contribute.html 4.24.3-0+++UE4+Release-4.24 518 0 Disabling core dumps. WARNING: Version mismatch detected: You are trying to connect to a simulator that might be incompatible with this API WARNING: Client API version = 0.9.10 WARNING: Simulator API version = c8a0b059 /home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32 warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow')) Traceback (most recent call last): File "oatomobile_test.py", line 7, in observation = environment.reset() File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/core/rl.py", line 67, in reset return self.simulator.reset(*args, *kwargs) File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/simulators/carla/simulator.py", line 1748, in reset obs = self.step(action=None) File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/simulators/carla/simulator.py", line 1780, in step timeout=defaults.QUEUE_TIMEOUT, File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/core/simulator.py", line 120, in get_observations return Observations(self.sensors, args, *kwargs) File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/core/simulator.py", line 82, in init for uuid, sensor in sensors.items()] File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/core/simulator.py", line 82, in for uuid, sensor in sensors.items()] File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/simulators/carla/simulator.py", line 1290, in get_observation destination=end_waypoint.transform.location, File "/home/yx/anaconda3/envs/carla_ros/lib/python3.7/site-packages/oatomobile/util/carla.py", line 740, in global_plan waypoints, roadoptions = zip(grp.trace_route(origin, destination)) File "/home/yx/carla/PythonAPI/carla/agents/navigation/global_route_planner.py", line 351, in trace_route route = self._path_search(origin, destination) File "/home/yx/carla/PythonAPI/carla/agents/navigation/global_route_planner.py", line 224, in _path_search start, end = self._localize(origin), self._localize(destination) File "/home/yx/carla/PythonAPI/carla/agents/navigation/global_route_planner.py", line 156, in _localize edge = self._road_id_to_edge[waypoint.road_id][waypoint.section_id][waypoint.lane_id] TypeError: 'NoneType' object is not subscriptable WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 207 (sensor.other.collision) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 208 (sensor.camera.semantic_segmentation) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 209 (sensor.camera.rgb) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 210 (sensor.camera.rgb) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 212 (sensor.lidar.ray_cast) terminate called without an active exception Aborted (core dumped)

Dashora7 commented 3 months ago

These errors are possibly related to the warning you've been given due to Simulator / Client version mismatch. I'd fix that first to ensure that warning isn't causing any of this

942411526 commented 3 months ago

These errors are possibly related to the warning you've been given due to Simulator / Client version mismatch. I'd fix that first to ensure that warning isn't causing any of this

Thank you very much, our OATomobile is working, but i have idea to run the vint in this can you released the code about deployment or give me some guidance?