search

nerfstudio-project / nerfstudio

A collaboration friendly studio for NeRFs
https://docs.nerf.studio
Apache License 2.0
9.34k stars 1.26k forks source link

Gopro MAX .360 format #1966

Open clementh44 opened 1 year ago

clementh44 commented 1 year ago

Hi, There seems to be an issue with the video format coming from a GoPro MAX. This article explains the problem.

File format: .360 Projection: EAC However, the projection is in two different streams, so the file requires some processing before it can be converted into an equirectangular projection.

There is a fork of ffmpeg available to handle this format.

Here's what the current ffmpeg command does: We can see that it only takes a single stream (missing the top and bottom of the original video, which is the second stream), and there are two lines where the pixels are duplicated. frame_00001

And here's the result of exporting an image from the video. GS012361-2

I have tried playing around with ffmpeg, but I'm not familiar with all the filters. For example, one can combine the two streams to create the equirectangular view. ffmpeg -i *.360 -filter_complex "[0:0]scale=iw:ih/2[top]; [0:1]scale=iw:ih/2[bottom]; [top][bottom]vstack[ou]; [ou]v360=eac:e" -r 0.1 frame_%05d.png frame_00001

clementh44 commented 1 year ago

I tried processing the images with ffmpeg, but it would be necessary to blend the duplicated parts of the image. Currently, it removes 64px from each side with: ffmpeg -i *.360 -filter_complex "[0:0][0:1]vstack,split=3[tp1][tp2][tp3]; [tp1]crop=w=688:x=0:y=0[left]; [tp2]crop=w=2592:x=752:y=0[middle]; [tp3]crop=w=688:x=3408:y=0[right]; [left][middle][right]hstack=inputs=3[out]; [out]scale=5376:3024,v360=eac:e" -r 0.1 frame_%05d.png

frame_00001

to be continued...

f-dy commented 1 year ago

Did you try https://github.com/trek-view/max2sphere ?

Also note that equirectangular images coming from 360 cameras will have issues, because it's physically impossible to build a 360 snapshot camera with a single optical center. The stitched 360 images from your GoPro MAX or insta 360 or Ricoh Theta may look nice, but they will have parallax issues, especially if your scene contains objects that are close. So don't expect much quality from the resulting nerf.

This should probably be mentioned in the doc https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#data

Probably a better way to use these images is to use the raw fisheye images rather than the stitched equirectangular images.

True panoramic devices use either a catadioptric system (no classical lens), or a rotating optical system (eg the classical DSLR-on-a-gimbal), so that all optical rays pass through the same optical center.

More info: https://cave.cs.columbia.edu/projects/categories/project?cid=Computational+Imaging&pid=Catadioptric+Cameras+for+360+Degree+Imaging