search

robodhruv / visualnav-transformer

Official code and checkpoint release for mobile robot foundation models: GNM, ViNT, and NoMaD.
http://general-navigation-models.github.io
MIT License
425 stars 56 forks source link

Collision Issues During Simulation #32

Open chungcode0218 opened 2 weeks ago

chungcode0218 commented 2 weeks ago

Dear Author,

I am currently facing an issue and would like to seek your advice. I am running the NoMaD explore simulation on CARLA, but after sending the linear and angular velocities outputted by pd_controller.py to the car in CARLA, collisions often occur. My waypoint visualization seems to be normal. I have a few questions:

  1. In deployment/config/robot.yaml, there are settings such as max_v: 0.2, max_w: 0.4, and frame_rate: 4. Should I set the maximum speed and maximum angular velocity of the car in CARLA to 0.2 and 0.4 as well (I tried this and found that the car's speed was too fast)? Or should these values be manually adjusted based on the actual control situation of the car?

  2. Each path generated by NoMaD has 8 waypoints. Should we track the nearest waypoint? Or should we track the 2nd or 3rd waypoint, depending on the situation? For instance, when there is an obstacle on the right side, my visualization shows that the entire path clearly shifts to the left. However, the car still slowly moves to the right, possibly because the first waypoint is too close. Thank you for your help.

Cxf123Coder commented 1 week ago

Did you encounter any obstacle avoidance problems during your testing? Looking forward to your reply

ajaysridhar0 commented 1 day ago

1) I recommend adjusting the values of max_v, max_w, and the rate in the config file based on what best controls the car. 2) You're right; the 1st waypoint is usually too close. Tracking the 3rd waypoint gave us the best results across platforms.

One thing to note about the NoMaD model is that it is only trained on real-world robotics data, so simulations such as CARLA appear out of distribution. If configuring the robot.yaml file does not fix the problem, I recommend fine-tuning the base policy with your CARLA dataset. For more information on how fine-tuning can improve the model's performance, we tried fine-tuning ViNT (another model in the family of general navigation models) on CARLA, and you can see our results in section 6.3 of the ViNT paper. Please let me know if you have any questions.