princeton-computational-imaging / NSF

Official code repository for the paper: "Neural Spline Fields for Burst Image Fusion and Layer Separation"
MIT License
280 stars 14 forks source link

Suggestions asking for inputting data #9

Open RoveFlemington opened 5 days ago

RoveFlemington commented 5 days ago

Are there any special suggestions for inputting burst images, such as should the camera movement/tremor avoid being too large?? Actually, in my testing, significant optical flow changes make my reconstructed background image blurry.

RoveFlemington commented 5 days ago

test_result

Ilya-Muromets commented 5 days ago

Absolutely, I kind of go over this in the supplemental section of the paper: https://light.princeton.edu/wp-content/uploads/2023/12/neural-spline-fields.pdf

But the gist of it is that NSF is a very simple stochastic optimization model, you're kind of relying on it falling into a local optima (separating the occlusion from the transmission) based on millions of small updates. If that optimum is too far away from the initialization (e.g., your camera moves a lot), then there's a good chance the model will never find it.

As a rule of thumb, it seems to do fairly well when the motion is on the scale of the occlusion (e.g., if you have a fence that's X pixels wide, then during the burst it should ideally move between X and 3X pixels up/down/left/right to be well reconstructed).

RoveFlemington commented 13 hours ago

Thanks a lot for your useful comment. I have another question here, that is, when I train the model, the GLIBC version of the server cannot match to the tiny-cuda-nn. Is it convenient for you to directly give {occlusion}'s weight data? I'll appreciate it much! My e-mail: zhangzhe21@tsinghua.org.cn

Ilya-Muromets commented 4 hours ago

I'm afraid you'll need to have tiny-cuda-nn working even to load the weights / test inference. Have you tried a prebuilt wheel (e.g., https://github.com/OutofAi/tiny-cuda-nn-wheels) ?