INVIGORATE is tested on: Ubuntu 18.04 ROS melodic Python 2.7
git clone https://github.com/AdaCompNUS/INVIGORATE.git
cd INVIGORATE
git submodule init
git submodule update --recursive --init
# clone another copy for use inside docker
git clone https://github.com/AdaCompNUS/INVIGORATE.git INVIGORATE_docker
cd INVIGORATE_docker
git submodule init
git submodule update --recursive --init
INVIGORATE relies on 4 different neural networks. Their environments have to be setup respectively. It is recommended that they each have their separate conda environment or docker. Since MAttNet and VMRN requires pytorch 0.4 and CUDA 8.0, it is recommended to use our provided docker to run them.
We use a custom version of detectron2. To install dependencies, do
conda create -n detectron2 python=3.8 # detectron uses python3
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
cd src/faster_rcnn_detector/detectron2
pip install -e .
pip install rospkg catkin_pkg opencv-python # to make ROS work with python3
Both MAttNet and VMRN requires pytorch 0.4 and cuda 9.0. You can setup the environment on your own or use our provided docker. Our docker uses cuda 9.0 and it has a conda environment called torch_old that contains pytorch 0.4.
cd .. # move to the parent folder of INVIGORATE
docker pull adacompnus/vmrd
docker run --gpus all -v "$(pwd)"/INVIGORATE_docker:/home/INVIGORATE_docker --network host -it adacompnus/vmrd /bin/bash # mount INVIGORATE_docker folder into docker and give GPU and network access
cd /home/INVIGORATE_docker # go to mounted folder
catkin build
source devel/setup.bash
conda activate torch_old # the docker has a conda environment with cuda 9.0 and pytorch 0.4
Proceed to follow the official instruction to setup an environment for MAttNet. Some of the most important steps are shown below.
conda activate torch_old # if you have not done this.
cd src/mattnet_server/MAttNet
cd pyutils/mask-faster-rcnn
git clone https://github.com/cocodataset/cocoapi data/coco
cd data/coco/PythonAPI
make
cd pyutils/mask-faster-rcnn/lib
make
cd pytuils/refer
make
We are using a legacy version of VMRN. Just go to vmrn_old submodule and run make.sh
conda activate torch_old # if you have not done this.
cd src/mrt_detector/vmrn_old
bash make.sh
docker run --gpus all --network host -it adacompnus/ingress /bin/bash
$ luarocks install cutorch
$ luarocks install cunn
$ luarocks install cudnn
Create a conda environment for INVIGORATE
conda create -n invigorate python=2.7
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
pip install -r requirements.txt
All models are stored here
Download model_final_cascade.pth
put it in
Download all_in_one_1_25_1407_0.pth
put it in
Download mattnet pretrained model on refCOCO from its official repository. (The root directory should be INVIGORATE_docker if you follow the procedure above.)
Included in the INGRESS docker
Download density estimation pickle files. (ground_density_estimation_mattnet.pkl and relation_density_estimation.pkl)
put it in
Download invigorate dataset from here.
Extract it and put it under the
This repo is a ROS workspace. Now build it.
catkin build
source devel/setup.bash
Assuming environments for different neural network modules are setup, launch them in separate terminals
roscore
conda activate <detectron2_env> # use your environment name
source devel/setup.bash
cd src
bash launch_detectron2.sh
conda activate torch_old
source devel/setup.bash
cd src
bash launch_mattnet.sh
conda activate torch_old
source devel/setup.bash
cd src
bash launch_vmrn.sh
docker run -it adacompnus/ingress:v1.2 /bin/bash
ingress
Now we are ready to run INVIGORATE! There are 100 scenarios in the provided dataset, we have included experiment results for scenario 1-10. To replicate a particular scenario,
conda activate invigorate
source devel/setup.bash
cd src/scripts
python dataset_demo --scene_num 1
If you want to generate captions:
python dataset_demo --scene_num 1 --captions
The result will be saved inside src/logs directory. The result produced should be equivalent to the result in the dataset.