Only tested on an iPhone 14 Pro. Older phones will probably struggle. You can adjust the render resolution when loading a model by increasing the rendererDownsample to 4x or more on the 'SplatModelInfo' struct.
:warning: This was a quick weekend hack project for educational purposes. As such the code is bad.
🔹 NOTE: PLY models use git LFS for file storage. You may need to run git lfs pull if PLYs are missing
Notes
Does not use a tile based renderer, just basic quad rasterization
Spherical harmonics are not supported
Sorting is done on main thread with std::sort on CPU
Todos
fix pan / drag / rotate gesture handling for moving the model - now it's broken
snap models to AR planes
GPU sorting
detect centroids and model extents on load rather than hand specifying
Parts of the code are based on the original Gaussian-Splatting software and is governed by the Gaussian-Splatting License, which can be found in the [LICENSE]() file in this repository. The original software was developed by Inria and MPII.
Please be advised that the software in this repository cannot be used for commercial purposes without explicit consent from the original licensors, Inria and MPII.
There are several blog posts on how to train your own models from a set of images.
The output PLY model is what you want to keep ( e.g. 'iteration_30000/point_cloud.ply' )
The included models were trained on google colab with an A100 GPU.
Models Attribution
Includes two models from the NeRF synthetic blender dataset
The renders are from modified blender models located on blendswap.com:
IMPORTANT: You must agree to not ever review or reference in any way any other implementations which have other licenses.
All CUDA code must be replaced with Metal. You can replace PyTorch with C++ to achieve higher speed if needed, but it is not required.
The code must achieve the same PSNR and training / rendering speeds as the PyTorch version linked above, when run on an M1 MacBook Pro.
9 minutes to train 7k iterations on Garden scene.
60fps rendering.
Recommended Equipment
You will need access to an M-series processor Mac with Metal support. It does not strictly need to be an M1 MacBook Pro, as we can test performance for you.
You will likely need access to an NVIDIA GPU to run the reference implementation. I may be able to provide remote access but it's best if you can provide your own.
高斯泼溅相关资料
关于iOS使用metal渲染模型的资料: https://github.com/laanlabs/metal-splats
AR Gaussian Splatting for iOS & Metal
About
A buggy + slow AR gaussian splat renderer for iOS + Metal. Based on several existing projects:
Uses Satin + Forge for AR + 3d viewer ( by @rezaali )
Only tested on an iPhone 14 Pro. Older phones will probably struggle. You can adjust the render resolution when loading a model by increasing the rendererDownsample to 4x or more on the 'SplatModelInfo' struct.
:warning: This was a quick weekend hack project for educational purposes. As such the code is bad.
🔹 NOTE: PLY models use git LFS for file storage. You may need to run
git lfs pullif PLYs are missingNotes
Todos
Demo Video
Watch on YouTube
video credit: Nike shoe model from Alex Carliera
License(s)
Parts of the code are based on the original Gaussian-Splatting software and is governed by the Gaussian-Splatting License, which can be found in the [LICENSE]() file in this repository. The original software was developed by Inria and MPII.
Please be advised that the software in this repository cannot be used for commercial purposes without explicit consent from the original licensors, Inria and MPII.
Satin + Forge are released under the MIT license. See LICENSE for details.
Train your own model
There are several blog posts on how to train your own models from a set of images. The output PLY model is what you want to keep ( e.g. 'iteration_30000/point_cloud.ply' ) The included models were trained on google colab with an A100 GPU.
Models Attribution
Includes two models from the NeRF synthetic blender dataset
The renders are from modified blender models located on blendswap.com:
Problem Description
The goal is to write a Metal port of 3D Gaussian Splatting for Real-Time Radiance Field Rendering, an incredible new 3D reconstruction technique.
And there's good news -- you can use this PyTorch implementation as a reference! https://github.com/WangFeng18/3d-gaussian-splatting
Here is the original academic paper: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Acceptance Criteria
IMPORTANT: You must agree to not ever review or reference in any way any other implementations which have other licenses.
All CUDA code must be replaced with Metal. You can replace PyTorch with C++ to achieve higher speed if needed, but it is not required. The code must achieve the same PSNR and training / rendering speeds as the PyTorch version linked above, when run on an M1 MacBook Pro.
9 minutes to train 7k iterations on Garden scene. 60fps rendering. Recommended Equipment
You will need access to an M-series processor Mac with Metal support. It does not strictly need to be an M1 MacBook Pro, as we can test performance for you. You will likely need access to an NVIDIA GPU to run the reference implementation. I may be able to provide remote access but it's best if you can provide your own.