seathru_NeRF implentation

[Jordan-Pierce]

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like NA

Describe alternatives you've considered NA

Additional context Hey nerfstudio, thanks again for creating this resource. I was wondering if you had seen or would consider adding seathru_NeRF as an additional method? Nothing better than standardized and consolidated code.

SeaThru is physics-based approach to dehaze images collected underwater, and required a SfM-type setup. Since NeRFs, the original authors implemented the technique as a NeRF implementation.

Links to repo, both the original paper, and the NeRF paper, and some media for your viewing pleasure. Cheers.

[maturk]

Hi @Jordan-Pierce, it has not been added as a method, however I know someone on the discord channel has been trying to implement the method using nerfstudio and has gotten pretty far with it, but it is still a work in progress :D It seems to be a bit challenging due to some differences between the google mip-nerf original code and nerfstudio.

[acse-pms122]

Hi @Jordan-Pierce!

I have implemented it (with a few deviations from the original paper and repo) and am getting comparable results on the SeaThru-NeRF dataset. I am happy to collaborate if you would like to look into it:)

[longhose]

Hi, I'm looking forward to the SeaThru-NeRF implementation. I have several green water photogrammetry datasets that I would love to try with SeaThru-NeRF.

[psetinek]

@longhose I finished a basic implementation, you can check it out here: https://github.com/AkerBP/seathru_nerf

[longhose]

@psetinek, thank you very much for sharing this implementation and for the clear installation instructions.

I initially ran into the CUDA_OUT_MEMORY_ERROR. I got Seathru_NeRF to run on my GTX 1065 4GB GPU by setting the following: --pipeline.datamanager.train-num-rays-per-batch 2048 --pipeline.datamanager.eval-num-rays-per-batch 512 --pipeline.datamanager.pixel-sampler.num-rays-per-batch 512

I'm testing it with a dataset from the green waters of Maple Bay on Vancouver Island BC, Canada. The results are slightly better than with nerfacto: better color saturation and more transparent water. But, there are still a lot floating ghosts.

https://github.com/nerfstudio-project/nerfstudio/assets/34402981/9ca78853-21bd-4d6d-bdc9-2e3041549161

[psetinek]

@longhose I would say the most critical hyperparameter to get right is the beta, as it controls the separation of medium and objects. Are you able to filter out the water effects? (when rendering out a video of J and not rgb?)

[Jordan-Pierce]

@psetinek well done! I'll test it out later this week and share some results here. Do you plan to make a PR with nerfstudio?

[longhose]

@psetinek I am able to filter J, rgb and bs renders. The differences between the j and rgb are visible but light despite a clear bs. I started playing with the beta hyperparameter. I see the default value is 100.0. What would be the valid range for this parameter?

[psetinek]

@Jordan-Pierce cool! let me know how it works for you! Yes, I am planning to make a PR so a documentation of the method is featured in nerfstudio.

[psetinek]

@longhose Hi, it seems like you have some separation between objects and medium, not a bad start! Concerning hyperparams: I got the best results for my dataset using this value. I have sadly not had the time to properly test it on different datasets. If you contact me directly (e.g. on discord: psetinek), I could send you my master's thesis report where I discuss some of the most important hyperparams and the experiments I tried:)

[Jordan-Pierce]

@longhose Hi, it seems like you have some separation between objects and medium, not a bad start! Concerning hyperparams: I got the best results for my dataset using this value. I have sadly not had the time to properly test it on different datasets. If you contact me directly (e.g. on discord: psetinek), I could send you my master's thesis report where I discuss some of the most important hyperparams and the experiments I tried:)

@psetinek, I'd like a look at the thesis, request via discord sent