GSGEN: Text-to-3D using Gaussian Splatting

This repository contains the official implementation of GSGEN: Text-to-3D using Gaussian Splattng.

Paper | Project Page

Video results

https://github.com/gsgen3d/gsgen/assets/44675551/6f3c0df7-e3d1-4a37-a617-8a35ade2d72d

https://github.com/gsgen3d/gsgen/assets/44675551/64e5662f-e8d5-4380-a2ac-540d6789f65b

https://github.com/gsgen3d/gsgen/assets/44675551/c9c25857-3b5c-4338-adb0-e8e0910c8260

https://github.com/gsgen3d/gsgen/assets/44675551/25e3a94e-5a3b-4e14-bdd5-bfbf44fc2b82

Instructions:

1. Install the requirements:

   pip install -r requirements.txt

2. Build the extension for Gaussian Splatting:

   cd gs
   ./build.sh

3. Start training!

   python main.py --config-name=base prompt.prompt="<prompt>"

   You can specify a different prompt for Point-E:

   python main.py --config-name=base prompt.prompt="<prompt>" init.prompt="<point-e prompt>"

Viewer

splat viewer

We support splat viewer now ! Click the captions of text-to-3D results in our project page to watch the assets in a WebGL based viwer. Example: a pineapple. This great viewer achieves > 40 FPS on my MacBook with M1 pro chip.

viser based viewer (Visualize checkpoints on your own computer)

Start the Viewer by:

python vis.py <path-to-ckpt> --port <port>

If you are training on servers, tunneling the port using SSH

ssh -L <your_local_port>:<your_server_ip>:<your_server_port> <your_username>@<your_server>

then open the viewer in your host computer on port <your_local_port>.

Exports

First set the PYTHONPATH env var:

export PYTHONPATH="."

To .ply file

python utils/export.py <your_ckpt> --type ply

To .splat file

python utils/export.py <your_ckpt> --type splat

To mesh (Currenly only support shape export)

python utils/export.py <your_ckpt> --type mesh --batch_size 65536 --reso 256 --K 200 --thresh 0.1

where the can be the path to the .pt checkpoint file or, more conveniently, can be the id for the run (the display name of the run in wandb, e.g. 0|213630|2023-10-11|a_high_quality_photo_of_a_corgi). The exported files are reside in the exports/<export-type>.

If you encounter troubles in exporting in colab, using -m will work:

python -m utils.export <your_ckpt> --type <export_type>

Updates

• [2023-10-08] Now support exports to .ply and .splat files. Mesh exporting are coming soon.
• [2023-10-13] Now support Shap-E initialize, try it with init.type="shap_e"

TODO

• [ ] Support full mesh export. (Coming soon)
• [ ] Support VSD loss. (The VSD code is already done, further tuning is on the way)
• [ ] Support more guidance, e.g. zero123, make-it-3d, ControlNet Openpose, etc.

Acknowledgement

This code base is built upon the following awesome open-source projects:

• Stable DreamFusion
• threestudio
• 3D Gaussian Splatting
• splat
• Point-E
• Shap-E
• Make-it-3D

Thank the authors for their remarkable job !