Initialization and Alignment
for Adversarial Texture Optimization

ECCV 2022

Initialization and Alignment for Adversarial Texture Optimization. ECCV 2022.
Xiaoming Zhao, Zhizhen Zhao, and Alexander G. Schwing.

Project Page | Paper

Table of Contents

• Environment Setup
• UofI Texture Scenes
• Compile TexInit
• Train and Evaluate UofI
• Train and Evaluate ScanNet
• Citation

Environment Setup

conda env create -f environment.yaml

We set the following environment variables for later usage:

cd /path/to/this/repo
CODE_ROOT=$PWD
export TEX_INIT_DIR=${CODE_ROOT}/advtex_init_align/tex_init

export THIRDPARTY_DIR=${CODE_ROOT}/third_parties
mkdir -p ${THIRDPARTY_DIR}

And we use scene_04 as an example in this README:

export SCENE_ID=scene_04

PyTorch3D

We rely on PyTorch3D for rendering and rasterization. Please follow the official instruction to build it from source:

conda install -c conda-forge -c fvcore fvcore
conda install -c bottler nvidiacub

cd ${THIRDPARTY_DIR}
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d && git checkout d07307a

pip install -e . --verbose

UofI Texture Scenes

We self-collect data with an iOS App developed based on ARKit (XCode 13.1). Please download the dataset from the release page or this link, uncompress it, and place it under ${TEX_DIR}/dataset/uofi. The structure should be:

.
+-- dataset
|  +-- uofi
|  |  +-- scene_01
|  |  +-- scene_02
|  |  ...

Data Reader

The data is in binary format and consists of information for RGB, depth, camera matrices, and mesh. We provide a Python script to read data from it. Run the following command:

export PYTHONPATH=${CODE_ROOT}:$PYTHONPATH && \
python ${CODE_ROOT}/advtex_init_align/data/bin_data_reader.py \
--stream_dir ${CODE_ROOT}/dataset/uofi/${SCENE_ID} \
--save_dir ${CODE_ROOT}/dataset/extracted/${SCENE_ID}

The extracted data will be saved with structure:

.
+-- dataset
|  +-- extracted
|  |  +-- scene_04
|  |  |  +-- mesh.ply  # file
|  |  |  +-- rgb       # folder
|  |  |  +-- depth     # folder
|  |  |  +-- mat       # folder

We provide a notebook to illustrate the format of the extracted data.

Compile TexInit

The code has been tested on Ubuntu 18.04 with GCC 7.5.0.

Install Dependencies

1. Install dependencies from Package Manager:

sudo apt-get install libmetis-dev libpng-dev libsuitesparse-dev libmpfr-dev libatlas-base-dev liblapack-dev libblas-dev

2. Manually install Eigen3.4:

   cd ${THIRDPARTY_DIR}
   wget https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip
   unzip eigen-3.4.zip

3. Manuall install Boost 1.75.0. Please follow the official instruction to install Boost:

   
   cd ${THIRDPARTY_DIR}
   wget https://boostorg.jfrog.io/artifactory/main/release/1.75.0/source/boost_1_75_0.tar.bz2
   tar --bzip2 -xf boost_1_75_0.tar.bz2

mkdir -p ${THIRDPARTY_DIR}/boost_1_75_0/build cd ${THIRDPARTY_DIR}/boost_1_75_0/ ./bootstrap.sh --prefix=${THIRDPARTY_DIR}/boost_1_75_0/build ./b2 install

4. Manually install [CGAL 5.1.5](https://github.com/CGAL/cgal). Please follow [the official instruction](https://doc.cgal.org/5.1.5/Manual/installation.html):
```bash
cd ${THIRDPARTY_DIR}
wget https://github.com/CGAL/cgal/releases/download/v5.1.5/CGAL-5.1.5.zip
unzip CGAL-5.1.5.zip
cd ${THIRDPARTY_DIR}/CGAL-5.1.5 && mkdir install && mkdir build && cd build
cmake -DCGAL_HEADER_ONLY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../install ..
make
make install

Compile

Modify the Makefile based on your setup:

• Set the variable THIRDPARTY_DIR to the value of ${THIRDPARTY_DIR};
• Set the variable CUDA_ROOT to your CUDA's root directory, e.g., /usr/local/cuda-11.1.

Then use the following command to compile TexInit execution file:

cd ${TEX_INIT_DIR}
make tex_init DEBUG=0 -j 8

Train and Evaluate UofI

Train

We provide run.sh to illusrtate the overall pipeline. The following command will generate texture in ${CODE_ROOT}/experiments/uofi/${SCENE_ID}/optimized_texture_test_1_10, which can be directly used in 3D rendering engine, e.g., Blender and MeshLab.

bash ${CODE_ROOT}/run.sh \
  ${CODE_ROOT} \
  ${SCENE_ID} \
  run_train

Evaluate

The following command will compute and save quantitative results in ${CODE_ROOT}/experiments/uofi/${SCENE_ID}/test_1_10/eval_results.

bash ${CODE_ROOT}/run.sh \
  ${CODE_ROOT} \
  ${SCENE_ID} \
  run_eval

Train and Evaluate ScanNet

Download ScanNet raw data from the website. Place them under ${TEX_DIR}/dataset/scannet_raw with structure:

.
+-- dataset
|  +-- scannet_raw
|  |  +-- scene0000_00
|  |  |  +-- scene0000_00.sens  # file
|  |  |  ...
|  |  +-- scene0001_00
...

We use scene0016_00 as an example:

export SCENE_ID=scene0016_00

Train

We provide run_scannet.sh to illusrtate the overall pipeline. The following command will generate texture in ${CODE_ROOT}/experiments/scannet/${SCENE_ID}/optimized_texture_test_1_10, which can be directly used in 3D rendering engine, e.g., Blender and MeshLab.

Note, since ScanNet's scenes contain thousands of high-resolution images, the processing time will be much longer than that of UofI Texture Scenes.

bash ${CODE_ROOT}/run_scannet.sh \
  ${CODE_ROOT} \
  ${SCENE_ID} \
  run_train

Evaluate

The following command will compute and save quantitative results in ${CODE_ROOT}/experiments/scannet/${SCENE_ID}/test_1_10/eval_results.

bash ${CODE_ROOT}/run_scannet.sh \
  ${CODE_ROOT} \
  ${SCENE_ID} \
  run_eval

Citation

Xiaoming Zhao, Zhizhen Zhao, and Alexander G. Schwing. Initialization and Alignment for Adversarial Texture Optimization. ECCV 2022.

@inproceedings{zhao-tex2022,
title = {Initialization and Alignment for Adversarial Texture Optimization},
author = {Xiaoming Zhao and Zhizhen Zhao and Alexander G. Schwing},
booktitle = {Proc. ECCV},
year = {2022},
}

Acknowledgements

• We build on Adversarial Texture Optimization.
• Mesh flattening code is adopted from boundary-first-flattening.
• Markov Random Fields solver comes from GSPEN.
• S3 metrics is adopted from author's implementation.
  • Since the author's website is down, I share an intact mirror of the original implementation.