search

nvidia-isaac / nvblox

A GPU-accelerated TSDF and ESDF library for robots equipped with RGB-D cameras.
Other
705 stars 75 forks source link

readme

nvblox

Signed Distance Functions (SDFs) on NVIDIA GPUs

A GPU SDF library which offers

Above we show reconstruction using data from the 3DMatch dataset, specifically the Sun3D mit_76_studyroom scene.

Table of Contents

Why nvblox?

Do we need another SDF library? That depends on your use case. If you're interested in:

Below we visualize slices through a distance function (ESDF):

How to use nvblox

How use nvblox depends on what you want to do.

Out-of-the-box Reconstruction/ROS 2 Interface

For users who would like to use nvblox in a robotic system or connect easily to a sensor, we suggest using our ROS 2 interface.

The ROS 2 interface includes examples which allow you to:

The ROS 2 interface downloads and builds the library contained in this repository during installation, so you don't need to clone and build this repository at all.

Public Datasets

If you would like to run nvblox on a public datasets, we include some executables for running reconstructions on 3DMatch, Replica, and Redwood datasets. Please see our tutorial on running these.

C++ Interface

If you want to build nvblox into a larger project, without ROS, or you would like to make modifications to nvblox's core reconstruction features, this repository contains the code you need. Our tutorial provides some brief details of how to interact with the reconstruction in c++.

Native Installation

If you want to build natively, please follow these instructions. Instructions for docker are further below.

Install dependencies

We depend on:

Build and run tests and benchmark

Build:

cd nvblox/nvblox
mkdir build
cd build
cmake .. && make

Run test and benchmark from build dir:

ctest

Run an example

In this example we fuse data from the 3DMatch dataset. First let's grab the dataset. Here I'm downloading it to my dataset folder ~/datasets/3dmatch.

wget http://vision.princeton.edu/projects/2016/3DMatch/downloads/rgbd-datasets/sun3d-mit_76_studyroom-76-1studyroom2.zip -P ~/datasets/3dmatch
unzip ~/datasets/3dmatch/sun3d-mit_76_studyroom-76-1studyroom2.zip -d ~/datasets/3dmatch

Navigate to and run the fuse_3dmatch binary. From the nvblox base folder run

cd nvblox/build/executables
./fuse_3dmatch ~/datasets/3dmatch/sun3d-mit_76_studyroom-76-1studyroom2/ mesh.ply

Once it's done we can view the output mesh using the Open3D viewer. Instructions for installing open3d-viewer can be found below.

Open3D mesh.ply

You should see a mesh of a room:

Docker

We have several dockerfiles (in the docker subfolder) which layer on top of one another for the following purposes:

We rely on nvidia docker. Install the NVIDIA Container Toolkit following the instructions on that website.

We use the GPU during build, not only at run time. In the default configuration the GPU is only used at at runtime. One must therefore set the default runtime. Add "default-runtime": "nvidia" to /etc/docker/daemon.json such that it looks like:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         }
    },
    "default-runtime": "nvidia"
}

Restart docker

sudo systemctl restart docker

Now Let's build Dockerfile.deps docker image. This image install contains our dependencies.

docker build -t nvblox_deps -f docker/Dockerfile.deps .

In case you are running this on the Jetson, substitute the dockerfile: docker/Dockerfile.jetson_deps

Now let's build the Dockerfile.build. This image layers on the last, and actually builds the nvblox library.

docker build -t nvblox -f docker/Dockerfile.build .

Now let's run the 3DMatch example inside the docker. Note there's some additional complexity in the docker run command such that we can forward X11 to the host (we're going to be view a reconstruction in a GUI). Note that visualization does not work over SSH connections. Run the container using:

xhost local:docker
docker run -it --net=host --env="DISPLAY" -v $HOME/.Xauthority:/root/.Xauthority:rw -v /tmp/.X11-unix:/tmp/.X11-unix:rw nvblox

Let's download a dataset and run the example (this is largely a repeat of "Run an example" above).

apt-get update
apt-get install unzip
wget http://vision.princeton.edu/projects/2016/3DMatch/downloads/rgbd-datasets/sun3d-mit_76_studyroom-76-1studyroom2.zip -P ~/datasets/3dmatch
unzip ~/datasets/3dmatch/sun3d-mit_76_studyroom-76-1studyroom2.zip -d ~/datasets/3dmatch
cd nvblox/nvblox/build/executables/
./fuse_3dmatch ~/datasets/3dmatch/sun3d-mit_76_studyroom-76-1studyroom2/ mesh.ply

Now let's visualize. From the same executable folder run:

apt-get install libgl1-mesa-glx libc++1 libc++1-10 libc++abi1-10 libglfw3 libpng16-16
wget https://github.com/isl-org/Open3D/releases/download/v0.13.0/open3d-app-0.13.0-Ubuntu_20.04.deb
dpkg -i open3d-app-0.13.0-Ubuntu_20.04.deb
Open3D mesh.ply

to visualize on the jetson see below.

Open3D on Jetson

Open3D is available pre-compiled for the jetson (details here). Install via pip:

sudo apt-get install python3-pip
sudo pip3 install open3d==0.16.0

If version 0.16.0 is not available you need to upgrade your pip with pip3 install -U pip. You may additionally need to add the upgraded pip version to your path.

View the mesh via:

open3d draw mesh.ply

Building for multiple GPU architectures

By default, the library builds ONLY for the compute capability (CC) of the machine it's being built on. To build binaries that can be used across multiple machines (i.e., pre-built binaries for CI, for example), you can use the CMAKE_CUDA_ARCHITECTURE flag and set it to a semicolon-separated list of architectures to support.

cmake .. '-DCMAKE_CUDA_ARCHITECTURES=75;72' -DCMAKE_INSTALL_PREFIX=../install/ && make -j8 && make install

Building redistributable binaries, with static dependencies

If you want to include nvblox in another CMake project, simply find_package(nvblox) should bring in the correct libraries and headers. However, if you want to include it in a different build system such as Bazel, you can see the instructions here.

License

This code is under an [open-source license](@ref license) (Apache 2.0). :)

Paper

If you find this library useful for your research, please consider citing our paper:

@misc{millane2024nvblox,
      title={nvblox: GPU-Accelerated Incremental Signed Distance Field Mapping},
      author={Alexander Millane and Helen Oleynikova and Emilie Wirbel and Remo Steiner and Vikram Ramasamy and David Tingdahl and Roland Siegwart},
      year={2024},
      eprint={2311.00626},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}