Is there any possibility to speed up the pipeline (e.g., with GPU?)

NVlabs / intrinsic3d

Intrinsic3D - High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (ICCV 2017)

https://vision.in.tum.de/_media/spezial/bib/maier2017intrinsic3d.pdf

BSD 3-Clause "New" or "Revised" License

451 stars 80 forks source link

Is there any possibility to speed up the pipeline (e.g., with GPU?) #5

Closed zhangxaochen closed 5 years ago

zhangxaochen commented 5 years ago

Hi @robmaier ,

I've successfully built the Intrinsic3D program, with OpenMP enabled,

I'm running the program on a Ubuntu 16.04, with i7-5930, 32GB memory (seems exactly the same as your environment).

currently the execution time for optimization with AppIntrinsic3D takes around 6~7 hours on the Lion sequence with the default command:

../../build/bin/AppIntrinsic3D -s=sensor.yml -i=intrinsic3d.yml

I've seen in README.md:

the Intrinsic3D approach is computationally very demanding w.r.t. runtime and memory

but I'm still curious whether your optimization could be sped up with GPUs?

robmaier commented 5 years ago

Hi @zhangxaochen ,

the entire pipeline could in principle also be ported to GPU completey. The main challenge is the large-scale joint optimization problem, which could either be implemented using a custom GPU solver (compare e.g. Zollhoefer et al 2015) or hopefully even using Opt.

zhangxaochen commented 5 years ago

That will be awesome~

I'm looking forward to the execution time being within a few minutes 👍

BTW, will this be implemented in recent future?

robmaier commented 5 years ago

At the moment there are no plans of developing a fully GPU based pipeline. We have to count on other graduate students or so who are eager to work on this ... :/

yf225 commented 1 year ago

@robmaier I am interested in helping port it to GPU. Wonder how should I get started?

Alternatively, does having more CPU cores help?

robmaier commented 1 year ago

Hi @yf225, that sounds great! :)

The slowest part of the pipeline is the actual optimisation, which is already running multi-threaded on the CPU inside Ceres solver (so more CPU cores should be exploited by Ceres already).

Implementing the Levenberg-Marquardt optimisation manually (with manually calculated explicit partial derivates etc) could already lead to some speed-up.
For some more severe speed-ups, the way to go would be to follow the direction of Zollhoher et al 2015 and implement a CUDA PCG solver on the GPU. They have several predecessor works in which they have used this kind of solver for real-time GPU optimization. Feel free to reach out via email to robert.maier at tum.de for more discussions.

yf225 commented 1 year ago

@robmaier Thanks! Just sent you an email about a lighting estimation error I am running into - would really appreciate your help on it and hopefully after I get more familiar / comfortable with the workflow I can start contributing to it.