Improved Pipeline - Githubissues

ErlerPhilipp commented 1 year ago

In the default pipeline with our test data (100 images), most of the computation time is spent with Meshing and Texturing. We can expect a significant speed-up by replacing some of the steps, roughly 25 min -> 10min. Also, there are newer methods available that should improve the quality.

This will be the improved pipeline: <html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns="http://www.w3.org/TR/REC-html40">

Step | Node Type | Duration | Critical for Duration | Critical for Quality | Comment -- | -- | -- | -- | -- | -- 1 | Camera Init | 1s | | | 1 | Feature Extraction | 1m | + | ++ | 1 | Image Matching | 1s | | | 1 | Feature Matching | 2m | ++ | + | 1 | Structure from Motion | 4m | ++++ | +++ | 1 | Prepare Dense Scene | 20s | | | 1 | Depth Map | 20s | | + | 1 | Depth Map Filter | 15s | | + | | Editor | | | +++ | 2 | SfM Transform | 10s? | | | 2 | Normal Estimation | 1m? | + | + | Hemisphere normals from camera poses 2 | Meshing | 2m | ++ | ++ | PPSurf 2 | Mesh Filtering | 30s | | | Mesh decimation (quadric?) 2 | UV Unwrap | 2m? | ++ | + | By Meshlab? 2 | Texturing | 3m? | +++ | ++ | Use Uvs from previous step

Changes are only in Step 2 (after editor):

Normal Estimation (low priority): The new meshing can easily be extended to use rough normals. AliceVision likely doesn't output a point cloud with normals. We can make them ourselves from the camera poses. The hemisphere is known and will help the reconstruction a lot.
Meshing: Use POCO or similar. Retrain with the P2S ABC var-noise dataset. Add data augmentation (outliers, scene-based). Fix POCO bugs (marching cubes indexing, global layer ignored, kd-tree workers).
Mesh Filtering: Since we're not working with AliceVision's Tetrahedral reconstruction anymore, we need a different filtering method. The best choice is probably Quadric Mesh Simplification (PyMeshlab).
UV Unwrap (low priority): AliceVision's Meshing includes an unwrapping step, which is quite slow and produces inefficient textures. An alternative could be an automatic unwrapping by PyMeshlab or Blender. AliceVision's Texturing node can stay but the parameters must be set so that it uses the existing UV coordinates.

ErlerPhilipp commented 1 year ago

Open Points: how to deal with texturing > dense SfM Data

ErlerPhilipp commented 1 year ago

more ideas here: https://github.com/alicevision/Meshroom/issues/2185

dcbishop commented 11 months ago

Might be worth checking out Neuralangelo if you haven't already.

ErlerPhilipp commented 11 months ago

Hi!

Yes, I already had a quick look at Neuralangelo. The results look great. However, it has 2 drawbacks:

It needs camera poses (from COLMAP). If we need to run COLMAP before, it will probably take too long in total.
In the Collab, they say that Neuralangelo alone takes ~2 hours on a T4 GPU (maybe 30 min on our RTX 3090). The new planned pipeline should be around 20 min. With all optimizations, we might even get down to ~10min in total.
The new NVIDIA license is a bit difficult. This project will stay non-commercial in the foreseeable future but keeping an option for commercial use would be good. For that, we would need a license, which might be too expensive.

While Neuralangelo sure looks interesting, I think that AliceVision is the best choice at the moment.

ErlerPhilipp commented 7 months ago

too much work for now

ErlerPhilipp / Pix2Model

Improved Pipeline #159