SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction
ReLER, CCAI, Zhejiang University
✉Corresponding Author
Figure 1. With just a single image, SIFU is capable of reconstructing a high-quality 3D clothed human model, making it well-suited for practical applications such as 3D printing and scene creation. At the heart of SIFU is a novel Side-view Conditioned Implicit Function, which is key to enhancing feature extraction and geometric precision. Furthermore, SIFU introduces a 3D Consistent Texture Refinement process, greatly improving texture quality and facilitating texture editing with the help of text-to-image diffusion models. Notably proficient in dealing with complex poses and loose clothing, SIFU stands out as an ideal solution for real-world applications.
# News
- **[2024/6/18]** Due to visa check problem, the author can not come to the conference center in person. We are sorry about this [sad][cry].
- **[2024/4/5]** Our paper has been accepted as **Highlight** (Top 11.9% of accepted papers)!
- **[2024/2/28]** We release the code of **geometry reconstruction**, including test and inference.
- **[2024/2/27]** SIFU has been accepted by **CVPR 2024**! See you in Seattle!
- **[2023/12/13]** We release the paper on [arXiv](https://arxiv.org/abs/2312.06704).
- **[2023/12/10]** We build the [Project Page](https://river-zhang.github.io/SIFU-projectpage/).
# Installation
- Ubuntu 20 / 18
- **CUDA=11.6 or 11.7 or 11.8, GPU Memory > 16GB**
- Python = 3.8
- PyTorch = 1.13.0 (official [Get Started](https://pytorch.org/get-started/locally/))
We thank @[levnikolaevich](https://github.com/levnikolaevich) and @[GuangtaoLyu](https://github.com/GuangtaoLyu) for provide valuable advice on the installation steps.
If you don't have conda or miniconda, please install that first:
```bash
sudo apt-get update && \
sudo apt-get upgrade -y && \
sudo apt-get install unzip libeigen3-dev ffmpeg build-essential nvidia-cuda-toolkit
mkdir -p ~/miniconda3 && \
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh && \
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 && \
rm -rf ~/miniconda3/miniconda.sh && \
~/miniconda3/bin/conda init bash && \
~/miniconda3/bin/conda init zsh
```
```bash
# close and reopen the shell
git clone https://github.com/River-Zhang/SIFU.git
sudo apt-get install libeigen3-dev ffmpeg
cd SIFU
conda env create -f environment.yaml
conda activate sifu
pip install -r requirements.txt
```
Please download the [checkpoint (google drive)](https://drive.google.com/file/d/13rNSmQI_VaMtwlMBSUaxEGybzJEl5KTi/view?usp=sharing) and place them in ./data/ckpt
Please follow [ICON](https://github.com/YuliangXiu/ICON/blob/master/docs/installation.md) to download the extra data, such as HPS and SMPL (using ```fetch_hps.sh``` and ```fetch_data.sh```). There may be missing files about SMPL, and you can download from [here](https://huggingface.co/lilpotat/pytorch3d/tree/main/smpl_data) and put them in /data/smpl_related/smpl_data/.
# Inference
```bash
python -m apps.infer -cfg ./configs/sifu.yaml -gpu 0 -in_dir ./examples -out_dir ./results -loop_smpl 100 -loop_cloth 200 -hps_type pixie
```
# Testing
```bash
# 1. Register at http://icon.is.tue.mpg.de/ or https://cape.is.tue.mpg.de/
# 2. Download CAPE testset
bash fetch_cape.sh
# evaluation
python -m apps.train -cfg ./configs/train/sifu.yaml -test
# TIP: the default "mcube_res" is 256 in apps/train.
```
# Texture Refinement Module
The code is available for download on [google drive](https://drive.google.com/file/d/1GOpo8enZTWsaWMn_liPnPNmkaUeNsqJk/view?usp=sharing). Please note that the current code structure may not be well-organized and may require some time to set up the environment. The author plans to reorganize it at their earliest convenience.
# Applications of SIFU
## Scene Building
## 3D Printing
## Texture Editing
## Animation
## In-the-wild Reconstruction
# Bibtex
If this work is helpful for your research, please consider citing the following BibTeX entry.
```
@InProceedings{Zhang_2024_CVPR,
author = {Zhang, Zechuan and Yang, Zongxin and Yang, Yi},
title = {SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {9936-9947}
}
```