This is not an officially supported Google product.

SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections

Project Page | Video | Paper | Scenes

Implementation for SAMURAI. A novel method which decomposes multiple coarsly posed images into shape, BRDF and illumination.

SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections
Mark Boss¹, Andreas Engelhardt¹, Abhishek Kar², Yuanzhen Li², Deqing Sun², Jonathan T. Barron², Hendrik P. A. Lensch¹, Varun Jampani^2
1University of Tübingen, ²Google Research

Setup

A conda environment is used for dependency management

conda env create -f environment.yml
conda activate samurai

In case new datasets should be processed, we also provide a bash script to setup the u2net:

./setup_envs.sh

Running

Download one of our test scenes and extract it to a folder. Then run:

python train_samurai.py --config configs/samurai/samurai.txt --datadir [DIR_TO_DATASET_FOLDER] --basedir [TRAIN_DIR] --expname [EXPERIMENT_NAME] --gpu [COMMA_SEPARATED_GPU_LIST]

Run Your Own Data

The process of running your own data first requires a simple folder with images. Given that the environment is fully set up, the preparation is done with prepare_dataset.sh [PATH_TO_FOLDER]. This automatically creates the expected folder structure.

For the initial poses, we have created a GUI for labeling which can be started with python -m dataset_quadrant_labeler. Here, we propse to enter the image folder and start labeling. Keybindings are shown in the GUI and enable fast labeling. When all images are labeled the pose json file is automatically saved at the correct path.

Evaluation

The train_samurai.py can be called with a --render_only flag and the --config flag pointing to the args.txt of the experiments folder.

Mesh extraction

For the mesh extraction a blender installation is required. The extract_samurai.py script can be used to perform the extraction automated. Here, again the --config flag pointing to the args.txt of the experiments folder is required. Addtionally, a --blender_path flag pointing to the blender executable is required. The --gpus flag can be used to set the specific gpu for extraction and baking.

Citation

@inproceedings{boss2022-samurai,
  title         = {{SAMURAI}: {S}hape {A}nd {M}aterial from {U}nconstrained {R}eal-world {A}rbitrary {I}mage collections},
  author        = {Boss, Mark and Engelhardt, Andreas and Kar, Abhishek and Li, Yuanzhen and Sun, Deqing and Barron, Jonathan T. and Lensch, Hendrik P.A. and Jampani, Varun},
  booktitle     = {Advances in Neural Information Processing Systems (NeurIPS)},
  year          = {2022},
}