search

facebookarchive / Surround360

Surround360 is Facebook's open source hardware and software for capturing stereoscopic 3D 360 video for VR. The repo contains hardware designs, as well as software for camera control and rendering.
Other
2.17k stars 581 forks source link

using 185 fisheye #73

Closed cvaldesmol closed 8 years ago

cvaldesmol commented 8 years ago

Hi there I want to use 185 fisheye in the side cameras. what modifications are required to get the render working?

fbriggs commented 8 years ago

We have partial support for fisheye lenses on side cameras now, and it requires specifying a particular set of values in the rig json file. An example file is attached. However, we do not currently have intrinsic calibration or rectification for side fisheye cameras, and it would require some additional code to support that.

7fish.json.zip

cvaldesmol commented 8 years ago

Great. Thanks. Are you working on the rectification code? something we can expect soon? For calibration I think we can get away assuming a perfect lense

fbriggs commented 8 years ago

We are working on it, but I can't give any estimate right now of when it will be ready. Fisheye lenses on the side won't work well without it, unfortunately (you will get left/right eye stereo inconsistencies).

mimiaofandisp commented 8 years ago

Will it be a problem to change your calibratecamera to fisheye::calibrate etc? opencv has a bunch of functions to support fisheye correction. Not sure if that will be enough or not?

fbriggs commented 8 years ago

I have tried the code in fisheye::calibrate and couldn't get it to work (it crashed), but you might have better luck. Even if it runs successfully, there would probably be some additional work, specifically to implement the part of the code that projects the side camera images from their original format to equirectangular, while using the fisheye intrinsic parameters.

mimiaofandisp commented 8 years ago

but I think you already have the function projectRectilinearToSpherical to do the spherical projection from rectified undistorted images to a sphere. is there any problem if i use this function for fisheye inputs?

fbriggs commented 8 years ago

it won't work to use projectRectilinearToSpherical on fisheye inputs. that only works on rectilinear inputs.

mimiaofandisp commented 8 years ago

huh, I mean after the fisheye correction and rectification, the fisheye inputs already became rectilinear format, then why cannot you use projectRectilinearToSpherical? I do not think we need to directly project fisheye to a sphere.

BTW, i read your rectilinearToSpherical function, and it seems you project each rectilinear image to a sphere with each camera center as the sphere center (const float r = sqrt(u_u + v_v + f*f);). i am not sure if i understood it correctly, if so, i kinda got confused. i thought we should project all images to a sphere with a common center.

fbriggs commented 8 years ago

"I do not think we need to directly project fisheye to a sphere." --> This might be the source of some confusion. You actually do need to project the fisheye images directly to a sphere, and it will not work to go through a rectilinear projection in the middle. The reason is, for a 180 degree fisheye lens, the rectlinear image would have to be infinitely large.

However, for fisheye/f-theta lenses with FOV less than 180, it could work, although near 180 degrees you will still need a very large rectilinear projection to avoid losing detail.

Regarding your second point, that is correct. We project all side camera images to a sphere with the same center. One way to look at this is to imagine that everything in the scene is very far away. Then it is equivalent to having all of the cameras at the same point. This also has a desirable property, that after projecting, objects that are far away have 0 optical flow between adjacent cameras. You can see this in the /flow_images folder (the overlap images).

mimiaofandisp commented 8 years ago

Thanks for the clarification. That makes sense. Currently I use gopros with 120 FOV, so that is not a problem for the moment.

For the spherical projection, if you capture near scenes like indoor videos, will this approximation be a problem? What if you use (camera arm+focal length) as the sphere radius?

fbriggs commented 8 years ago

Everything is projected such that things have 0 flow when they are at a distance of infinity. However, if there are closer than infinity, the stitcher can still do it, it just means that it has to find a flow field that isn't all 0's. The closer things are, the larger the displacement needed in the flow fields, and lower the chance the flow solver will find a good solution.