Im2Avatar: Colorful 3D Reconstruction from a Single Image

This work reconstructs colorful 3D model from a single-view image. We tested the proposed framework on ShapeNet sub-database and selected 3D human meshes from MakeHuman.

[Project] [Paper]

Data

We provided processed ground truth 3D color and 2D-to-3D appearance flow data. The data will be automatically downloaded for the first time running training script. Both color data and flow data are stored as 3D volumes, but with different channels (color data has 3 channels and flow data owns 2 channels).

• Color data ranges from 0 to 1 to represent normalized colors for occupied surface voxels, and uses -1 for empty voxels.
• Flow data ranges from 0 to 1 to represent normalized coordinates in the 2D images for occupied surface voxels, and uses -1 for empty voxels.

For each 3D model, 12 images are provided from different viewpoints. The data indices used for training, validating and testing can be found in the data_list folder. Specifically, the processed ShapeNet subdataset can be downloaded from here, and Colorful Human dataset can be downloaded from here.

Requirements

• TensorFlow

ShapeNet Subdataset

Train

The model is trained per category, change the category id when working on different categories.

• Run shape training script for car category:

  python train_shape.py --cat_id 02958343

• Run color training script for car category:

  python train_color.py --cat_id 02958343

Inference

• To generate testing shape volumes for car category:

  python inference_shape.py --cat_id 02958343

• To generate testing color volumes for car category:

  python inference_color.py --cat_id 02958343

Eval

After generating all the shape and color volumes, evaluate surface PSNR and IoU. Please change the category id correspondingly within each file!

• Evaluate surface PSNR:

  python eval_surface_psnr.py

• Evaluate IoU:

  python eval_iou.py

Colorful Human Dataset

Train

• Run shape training script:

  python train_shape_human.py

• Run color training script:

  python train_color_human.py

Inference

• To generate testing shape volumes:

  python inference_shape_human.py

• To generate testing color volumes:

  python inference_color_human.py

Citation

Please cite this paper if you want to use it in your work,

@article{sun2018im2avatar,
  title={Im2Avatar: Colorful 3D Reconstruction from a Single Image},
  author={Sun, Yongbin and Liu, Ziwei and Wang, Yue and Sarma, Sanjay E},
  journal={arXiv preprint arXiv:1804.06375},
  year={2018}
}

License

MIT License