Features • Citing • Quick Start • Setup • Demos • Development
This package implements the Wild Visual Navigation (WVN) system presented in Frey & Mattamala et al. "Fast Traversability Estimation for Wild Visual Navigation" (2023) and later extended in Mattamala & Frey et al. "Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision" (2024). It implements a visual, self-supervised traversability estimation system for mobile robots, trained online after a few minutes of human demonstrations in the field.
elevation_mapping_cupy
@INPROCEEDINGS{frey23fast,
AUTHOR = {Jonas Frey AND Matias Mattamala AND Nived Chebrolu AND Cesar Cadena AND Maurice Fallon AND Marco Hutter},
TITLE = {{Fast Traversability Estimation for Wild Visual Navigation}},
BOOKTITLE = {Proceedings of Robotics: Science and Systems},
YEAR = {2023},
ADDRESS = {Daegu, Republic of Korea},
MONTH = {July},
DOI = {10.15607/RSS.2023.XIX.054}
}
If you are also building up on the STEGO integration or using the pre-trained models for comparison, please cite:
@INPROCEEDINGS{mattamala24wild,
AUTHOR = {Jonas Frey AND Matias Mattamala AND Libera Piotr AND Nived Chebrolu AND Cesar Cadena AND Georg Martius AND Marco Hutter AND Maurice Fallon},
TITLE = {{Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision}},
BOOKTITLE = {under review for Autonomous Robots},
YEAR = {2024}
}
If you are using the elevation_mapping_cupy
integration:
@INPROCEEDINGS{erni23mem,
AUTHOR={Erni, Gian and Frey, Jonas and Miki, Takahiro and Mattamala, Matias and Hutter, Marco},
TITLE={{MEM: Multi-Modal Elevation Mapping for Robotics and Learning}},
BOOKTITLE={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
YEAR={2023},
PAGES={11011-11018},
DOI={10.1109/IROS55552.2023.10342108}
}
We prepared a quick-start demo using a simulated Jackal robot. The demo runs on Docker, so no system dependencies are required. Please check the full instructions here
We recommend following the aforementioned Docker instructions as well as inspecting the Dockerfile for a clean, system-independent setup.
Otherwise, the next steps provide specific instructions to setup WVN on different systems.
The next steps assume you have the following hardware & software setup.
These are the minimum requirements to use the WVN scripts (no robot integration).
First clone the WVN and our STEGO reimplementation.
mkdir ~/git && cd ~/git
git clone git@github.com:leggedrobotics/wild_visual_navigation.git
git clone git@github.com:leggedrobotics/self_supervised_segmentation.git
./self_supervised_segmentation/models/download_pretrained.sh
(Recommended) Create new virtual environment.
mkdir ~/.venv
python -m venv ~/venv/wvn
source ~/venv/wvn/bin/activate
Install the wild_visual_navigation package.
cd ~/git
pip3 install -e ./wild_visual_navigation
pip3 install -e ./self_supervised_segmentation
Please refer to the Demos section below.
The following steps are required for a full installation, including the deployment tools. This enables the appropriate use of the ANYmal rosbags, enabling the robot model visualization and other deployment tools.
# Create new catkin workspace
source /opt/ros/noetic/setup.bash
mkdir -r ~/catkin_ws/src && cd ~/catkin_ws/src
catkin init
catkin config --extend /opt/ros/noetic
catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
# Clone repos
git clone git@github.com:ANYbotics/anymal_d_simple_description.git
git clone git@github.com:ori-drs/procman_ros.git
# Symlink WVN-repository
ln -s ~/git/wild_visual_navigation ~/catkin_ws/src
# Dependencies
rosdep install -ryi --from-paths . --ignore-src
# Build
cd ~/catkin_ws
catkin build anymal_d_simple_description
catkin build wild_visual_navigation_ros
# Source
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash
After successfully building the ROS workspace, you can run the entire pipeline by either using the launch file or by running the nodes individually. Open multiple terminals and run the following commands:
Run wild_visual_navigation
roslaunch wild_visual_navigation_ros wild_visual_navigation.launch
(ANYmal replay only) Load ANYmal description for RViZ
roslaunch anymal_d_simple_description load.launch
(ANYmal replay only) Replay Rosbag:
robag play --clock path_to_mission/*.bag
RViz:
roslaunch wild_visual_navigation_ros view.launch
Debugging (sometimes it is desirable to run the two nodes separately):
python wild_visual_navigation_ros/scripts/wvn_feature_extractor_node.py
python wild_visual_navigation_ros/scripts/wvn_learning_node.py
The general configuration files can be found under: wild_visual_navigation/cfg/experiment_params.py
This configuration is used in the offline-model-training
and in the online-ros
mode.
When running the online-ros
mode, additional configurations for the individual nodes are defined in wild_visual_navigation/cfg/ros_params.py
.
These configuration file is filled based on the ROS parameter server during runtime.
The default values for this configuration can be found under wild_visual_navigation/wild_visual_navigation_ros/config/wild_visual_navigation/default.yaml
.
We provide the python3 quick_start.py
script to inference traversability from images within the input folder (assets/demo_data/*.png
), given a pre-trained model checkpoint (assets/checkpoints/model_name.pt
, you can obtain them from Google Drive).
The script stores the result in the provided output folder (results/demo_data/*.png
).
python3 quick_start.py
# python3 quick_start.py --help for more CLI information
# usage: quick_start.py [-h] [--model_name MODEL_NAME] [--input_image_folder INPUT_IMAGE_FOLDER]
# [--output_folder_name OUTPUT_FOLDER_NAME] [--network_input_image_height NETWORK_INPUT_IMAGE_HEIGHT]
# [--network_input_image_width NETWORK_INPUT_IMAGE_WIDTH] [--segmentation_type {slic,grid,random,stego}]
# [--feature_type {dino,dinov2,stego}] [--dino_patch_size {8,16}] [--dino_backbone {vit_small}]
# [--slic_num_components SLIC_NUM_COMPONENTS] [--compute_confidence] [--no-compute_confidence]
# [--prediction_per_pixel] [--no-prediction_per_pixel]
To quickly test out the online training and adaption we provide some example rosbags (GDrive), collected with our ANYmal D robot. These can be tested using the ROS instructions
Here we provide some examples for the different sequences:
Lastly, we provide some general guidelines for development. These could be useful to test WVN with your own robot platform.
The WVN repo is structured in different folders, which we explain in the following figure:
📦wild_visual_navigation
┣ 📂assets
┣ 📂demo_data # Example images
┣ 🖼 example_images.png
┗ ....
┗ 📂checkpoints # Pre-trained model checkpoints (must be downloaded from Google Drive)
┣ 📜 mountain_bike_trail_v2.pt
┗ ....
┣ 📂docker # Quick start docker container
┣ 📂results
┣ 📂test
┣ 📂wild_visual_navigation # Core, ROS-independent implementation of WVN
┣ 📂wild_visual_navigation_anymal # ROS1 ANYmal helper package
┣ 📂wild_visual_navigation_jackal # ROS1 Jackal simulation example
┣ 📂wild_visual_navigation_msgs # ROS1 message definitions
┣ 📂wild_visual_navigation_ros # ROS1 nodes for running WVN
┗ 📂scripts
┗ 📜 wvn_feature_extractor_node.py # Main process for feature extraction and inference
┗ 📜 wvn_learning_node.py # Main process that generates supervision signals and the online training loop
┗ 📜 quick_start.py # Inference demo from pre-trained checkpoints
We recommend making a new ROS package to implement the overlays to run WVN with your own robot platform. We suggest inspecting the wild_visual_navigation_jackal as a reference.
In a nutshell, you need to configure: