Closed JonasFrey96 closed 6 months ago
This may be the highest increase in performance we can achieve. Create a demo Python node in which we just run DINO nothing else and measure the throughput. This would simulate the maximum inference we can achieve independent of all learning things.
Did a first experiment. Wrote a feature extraction node which is capable to run at 22Hz on my laptop. The forward pass on the left takes roughly 30ms of the callback function the output frequency when just publishing a single float value to measure the performance of the node is a stable 22.5Hz. The input frequency of the images are set to 115 Hz and currently compressed images.
Interestingly the bottleneck is moving the image onto the GPU and resizing it. Could we reconfigure the Alphasense driver to publish a downscaled version of the image:
If we integrate this change we can easily run all 3 cameras if we split into a hot path feature extraction node and learning node.
That's interesting and looks really promising. Is the code somewhere I can take a look?
A few comments:
std_msgs/Image
to torch.tensor
on the GPU?I will push it to a branch inheriting from devel in a moment. This considers the full resolution of std_msgs/Image which is 1080 x 1440 this we then at first have to convert to opencv then to torch (GPU) and then we rescale to (224,224).
Okay so now the idea would be:
Proprioception to create the supervision graph as we had Extracted Features and Segmentation Mask (synchronized, or here this could be CostumeMessage consisting of the MultiArray + Int32 Image )
Visualization of the path MLP weights
Code under development here: https://github.com/leggedrobotics/wild_visual_navigation/tree/dev/two_node_solution
TODOs
done
Maybe also run the small MLP which weights are synced from our WVN learning code allowing for low latency.
wild_visual_navigation_ros
Responsibility: training and graph things
MLP <- segment masks, features -> weights
wild_visual_navigation_runtime
Responsibility: Feature Extraction and Output publishing
<- 3x camera images <- MLP weights and threshold -> segment masks, features -> traversability, confidence output