Using vint on own robot, at starting position the closest_node keeps changing even though the robot is at standstill

robodhruv / visualnav-transformer

Official code and checkpoint release for mobile robot foundation models: GNM, ViNT, and NoMaD.

http://general-navigation-models.github.io

MIT License

425 stars 56 forks source link

Using vint on own robot, at starting position the closest_node keeps changing even though the robot is at standstill #19

Closed vedant-banerjee closed 4 months ago

vedant-banerjee commented 4 months ago

The closest_node at starting point, keeps flickering between 0 and 5 , and when the controller is run, the robot navigates but the closest node detected seems to be wrong , hence the robot is unable to find the goal node and stop, any help on how to debug this issue will be appreciated, even the start and end values are changing even though the robot is at standstill.

robodhruv commented 4 months ago

Hi Vedant,

That is not expected behavior and we're happy to help debug. Would it be possible for you to share a few images from the start, or the entire topomap if possible? What camera are you using?

Dhruv

vedant-banerjee commented 4 months ago

Hi Dhruv , The current update is i have tried fine tuning the model by training it and the issue of finding the closest_node and goal node still seems to be an issue, also my robot is a much larger size compared to the locobot , and there are collisions taking place if obstacles are placed on the path. The camera i am using is Intel Realsense D435I , i have recorded a bag with a small path , which goes only straight and i am trying to use that. I will share the images from the topomap along with my robot image.

vedant-banerjee commented 4 months ago

This is one of the topomaps, but i have tried with a few others as well, and issue of finding closest node and goal node is still there, and collisions are happening. @robodhruv

robodhruv commented 4 months ago

Thanks for sharing, Vedant! I am quite confident that the issue arises from the camera -- RealSense D435i is a fairly narrow FoV (~90 degrees) and quite OOD from what the model is trained on. I would recommend switching to a wide-angle camera, either to replace the D435i, or just add a second USB camera to your build. While this issue is particularly pronounced with ViNT/NoMaD, in general, I expect fisheye cameras to be more useful because the added FoV helps in short-range collision detection and avoidance.

Here is an example camera that you can use: https://www.amazon.com/ELP-Fisheye-Lightburn-Machine-Embedded/dp/B00LQ854AG/

vedant-banerjee commented 4 months ago

Alright thanks for the help, also does the robot size also matter? or this is taken into consideration while fine tuning the model, to avoid collisions @robodhruv

robodhruv commented 4 months ago

The robot size does matter, and can get you in trouble if the robot is too large, but I suspect this will not be a big issue for you unless you are trying to work in a very dense indoor environment. We have tested the model on an AgileX Tracer robot base, which is quite large (50x60 cm) and the model seems to work fine.

vedant-banerjee commented 4 months ago

Okay thanks i will change the camera as suggested and see if i get better results

robodhruv commented 4 months ago

Good luck and feel free to re-open the issue after you've tried!

vedant-banerjee commented 3 months ago

@robodhruv I have tried using a fisheye wide angle lens camera of fov 190 degree, i will share some images of the topomap, i am facing the same issue where the closest_node is not updating correctly as the robot is moving. and hence its losing track. I have not trained/fine tuned it, this is when i run zero shot.