search

SharkWipf / nerf_dataset_preprocessing_helper

Intelligent(?) image sharpness filtering for optimized NeRF training datasets
MIT License
62 stars 4 forks source link

Feature Request: support 360 video and images #1

Closed ihorizons2022 closed 1 year ago

ihorizons2022 commented 1 year ago

very handy tools, thank you very much. if the tool can support more camera types, it would be great.

SharkWipf commented 1 year ago

It does already support images (though the documentation might need some work), if you point the first script at a folder instead of a video, it will use the images inside as frames, in alphanumeric order. Currently it only checks for .jpg files, I'll change that to include other formats as well later today.

As for 360 video, I don't think I can easily replicate the image extraction from that without ballooning the codebase, but if you extract all frames through another means (i.e. ns-process-data) and then point this tool at the images folder generated, you should still be able to run it.

Of course, as long as there is a transforms.json, you should be able to run the second, post-colmap script in any case. Though I don't know how it'll handle sharpness-over-time with 360 images, you might need to tweak the settings a bit to make it work just right.

I don't use 360 video myself, so I'm not too familiar with it, if there's anything specific that I'm missing feel free to let me know.

ihorizons2022 commented 1 year ago

Thank you for your reply. Actually, maybe only one more step needed to process 360 video or image in your pipeline. After ffmpeg 360 video into equirectangular images, split these images into undistorted images. if the tool supports fisheye and equirectangular cameras, Nothing is better than good.

https://github.com/alicevision/Meshroom/blob/develop/meshroom/nodes/aliceVision/Split360Images.py

ihorizons2022 commented 1 year ago

IMHO，360 video will be the infrastructure for NERF and 3DGS in the near future.

SharkWipf commented 1 year ago

I might look into adding a separate workflow for equirects/360 videos some time (probably as a new 01_filter_360_video.py file or something), but I can't promise anything, unfortunately I'm quite busy with other things atm, and it's not a workflow I personally use or can easily test.
I simply don't have the time to invest in adding that workflow currently.
I'll be happy to look at any PRs though.

For now I've added support for more file extensions for the from-images flow, by default jpg,jpeg,png, but with an --exts flag that lets you specify anything else. Anything supported by OpenCV should work.

IMHO，360 video will be the infrastructure for NERF and 3DGS in the near future.

I agree they will probably play a big role on the professional side, however personally I'm more interested in the ability to turn any videos shot from anyone with any smartphone or camera into a good NeRF/3DGS. That not only puts NeRFs/3DGS in the hands of everyone with a camera without any special training, it would also allow for any old footage to be turned into NeRFs.
I also think this will be more interesting for companies once the technology further matures. Want to try out a new kitchen? Simply have your customer shoot a quick video of their current kitchen, do some software magic, and you can generate a NeRF/3DGS with a completely new kitchen in-situ, with an AR viewer so they can walk around right in their would-be-new kitchen in their own house.

But to get to that point, getting good NeRFs from bad videos needs to become much easier, and that's what I was hoping to help achieve with this script.

ihorizons2022 commented 1 year ago

Thank you for the idea of universal benefit. looking forward to any progress.