mujuco-py
and install the following apt packages if using Ubuntu:
sudo apt install -y libosmesa6-dev libgl1-mesa-glx libglfw3 patchelf
conda create -n uvd python==3.9 -y && conda activate uvd
git clone https://github.com/facebookresearch/vip.git
cd vip && pip install -e .
python -c "from vip import load_vip; vip = load_vip()"
```commandline git clone https://github.com/facebookresearch/r3m.git cd r3m && pip install -e . python -c "from r3m import load_r3m; r3m = load_r3m('resnet50')" ```
```commandline git clone https://github.com/penn-pal-lab/LIV.git cd LIV && pip install -e . && cd liv/models/clip && pip install -e . python -c "from liv import load_liv; liv = load_liv()" ```
```commandline git clone https://github.com/facebookresearch/eai-vc.git cd eai-vc && pip install -e vc_models ```
pip install -e .
We provide a simple API for decompose RGB videos:
import torch
import uvd
# (N sub-goals, *video frame shape)
subgoals = uvd.get_uvd_subgoals(
"/PATH/TO/VIDEO.*", # video filename or (L, *video frame shape) video numpy array
preprocessor_name="vip", # Literal["vip", "r3m", "liv", "clip", "vc1", "dinov2"]
device="cuda" if torch.cuda.is_available() else "cpu", # device for loading frozen preprocessor
return_indices=False, # True if only want the list of subgoal timesteps
)
or run
python demo.py
to host a Gradio demo locally with different choices of visual representations.
We post-processed the data released from original Relay-Policy-Learning that keeps the successful trajectories only and adapt the control and observations used in our paper by:
python datasets/data_gen.py raw_data_path=/PATH/TO/RAW_DATA
Also consider to force set Builder = LinuxCPUExtensionBuilder
to Builder = LinuxGPUExtensionBuilder
in PATH/TO/CONDA/envs/uvd/lib/python3.9/site-packages/mujoco_py/builder.py
to enable (multi-)GPU acceleration.
Since UVD's goal is to be an off-the-shelf method applying to any existing policy learning frameworks and models, across BC and RL, we provide minimal scripts for benchmarking the runtime showing negligible runtime under ./scripts
directory:
python scripts/benchmark_decomp.py /PATH/TO/VIDEO
and passing --preprocessor_name
with other preprocessors (default vip
) and --n
for the number of repeated iterations (default 100
).
For inference or rollouts, we benchmark the runtime by
python scripts/benchmark_inference.py
and passing --policy
for using MLP or causal GPT policy; --preprocessor_name
with other preprocessors (default vip
); --use_uvd
as boolean arg for whether using UVD or no decomposition (i.e. final goal conditioned); and --n
for the number of repeated iterations (default 100
). The default episode horizon is set to 300. We found that running in the terminal would be almost 2s slower every episode than directly running with python IDE (e.g. PyCharm, under the script directory and run as script instead of module), but the general trend that including UVD introduces negligible extra runtime still holds true.
If you find this project useful in your research, please consider citing:
@inproceedings{zhang2024universal,
title={Universal visual decomposer: Long-horizon manipulation made easy},
author={Zhang, Zichen and Li, Yunshuang and Bastani, Osbert and Gupta, Abhishek and Jayaraman, Dinesh and Ma, Yecheng Jason and Weihs, Luca},
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
pages={6973--6980},
year={2024},
organization={IEEE}
}