Equidistant fisheye model

[qume]

Using Equidistant fisheye model (equidistant_r3) always seems to fail at this line: https://github.com/alicevision/AliceVision/blob/bc7c981c31a0f119da37f1f6ff1865f09bbaf679/src/aliceVision/sfm/pipeline/sequential/ReconstructionEngine_sequentialSfM.cpp#L1205

Has this been run successfully? And if so is there a trick to getting past this error?

@servantftechnicolor we would very much like to use this, as the standard fisheye models based on pinhole model don't seem to be adequate, and this looks like a needed improvement for wide fisheyes/

[servantftechnicolor]

Hi,

Actually, the current SFM pipeline is restricted to pinhole cameras (Equidistant lens has been originally added for the panorama pipeline).

We are working on removing this restriction.

Could you elaborate on what is it needed for ? 3D reconstruction ?

Fabien

From: Luke Reid @.> Sent: Wednesday, December 20, 2023 2:30 AM To: alicevision/AliceVision @.> Cc: Fabien Servant @.>; Mention @.> Subject: [alicevision/AliceVision] Equidistant fisheye model (Issue #1631)

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Using Equidistant fisheye model (equidistant_r3) always seems to fail at this line: https://github.com/alicevision/AliceVision/blob/bc7c981c31a0f119da37f1f6ff1865f09bbaf679/src/aliceVision/sfm/pipeline/sequential/ReconstructionEngine_sequentialSfM.cpp#L1205https://url.avanan.click/v2/___https:/github.com/alicevision/AliceVision/blob/bc7c981c31a0f119da37f1f6ff1865f09bbaf679/src/aliceVision/sfm/pipeline/sequential/ReconstructionEngine_sequentialSfM.cpp%23L1205___.YXAzOnRlY2huaWNvbG9yOmE6bzpjNmQxNzBjMzYxOGEwMDQ3Y2VmMWE1ZmFhMjBmMGFhYTo2OjZiMDY6NWM4OTc0ZDE1MGQ1MGM3ZDI4NzkyNjczNDRhNmM1NTZiZTZmZWEzZDgyZDdiNTdlNjRmYzYyYmY2MmZhNTNhNDpoOlQ

Has this been run successfully? And if so is there a trick to getting past this error?

@servantftechnicolorhttps://url.avanan.click/v2/___https:/github.com/servantftechnicolor___.YXAzOnRlY2huaWNvbG9yOmE6bzpjNmQxNzBjMzYxOGEwMDQ3Y2VmMWE1ZmFhMjBmMGFhYTo2OjUzYWY6YjAxNmVmNjIxZDAzNWUwMmIxNTk4Njc4YjRkOTc3ODI1ODk2ZDRhOGM5Y2MzYjg3OGU4NTUyZDExMWQ1ODFmMzpoOlQ we would very much like to use this, as the standard fisheye models based on pinhole model don't seem to be adequate, and this looks like a needed improvement for wide fisheyes/

— Reply to this email directly, view it on GitHubhttps://url.avanan.click/v2/___https:/github.com/alicevision/AliceVision/issues/1631___.YXAzOnRlY2huaWNvbG9yOmE6bzpjNmQxNzBjMzYxOGEwMDQ3Y2VmMWE1ZmFhMjBmMGFhYTo2OmYzNjA6YjQzYjE3N2Q1YjUzZTYwY2I0MWFmNjM3MDI1ZjNjYWFlYTc5Njg1ZTljZGVmNTU3ODgyZDAyNjU0MjM3NzM5ZjpoOlQ, or unsubscribehttps://url.avanan.click/v2/___https:/github.com/notifications/unsubscribe-auth/AX5TBH7KZK4SMEEU7WQTXA3YKI5TJAVCNFSM6AAAAABA4BYUU2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2DSNRWG43TINI___.YXAzOnRlY2huaWNvbG9yOmE6bzpjNmQxNzBjMzYxOGEwMDQ3Y2VmMWE1ZmFhMjBmMGFhYTo2OjU1ZTA6YmUyNmNhODQ1MjU2MzI2MzI1NWYwNzdhN2Q3YjA0ZjUxZDIyOGM5MmJlZTYwNDAzNDQ4N2I5OGVhYzEzNjlkYzpoOlQ. You are receiving this because you were mentioned.Message ID: @.***>

[qume]

First, apologies for the very slow reply to your message. We have been doing some experiments to see if we can get a better idea of the issue we are having.

In a nutshell - we have tested doing mesh generation using (almost) all of the commercial packages out there, plus meshroom, and with very wide fisheyes, we are unable to get meshes which are of the same quality when using rectilinar lenses.

We suspected it's a limitation with the pinhole model, which is why I was excited to see different models in Meshroom.

The meshes when using fisheyes seem to end up with what should be hard/straight edges being 'wavy', and just generally poor reconstruction all around.

One thing we also suspect, which may not be fixed by just a non-pinhole model, is that we think the distortion is not equal / circular all the way around the lens. So there may need to be some distortion parameters which are a function of the polar angle around the principle point.

I would love to know if anyone else has managed to produce high quality meshes using fisheye lenses. We produce a 360 degree 3x12mp global shutter fisheye rig, and are keen to improve the quality of the meshes it produces.

[servantftechnicolor]

If you have a rig, and always use the same fisheye cameras, then you really should print / display calibration checkerboards and calibrate your lens for distortion. make sure your(s) boards cover the whole image, or have multiple images where all the regions of the images are measured. For me that's really the first step. Then You should calibrate the geometry of your rig (if it is fixed).

[NeedsMoar]

One thing we also suspect, which may not be fixed by just a non-pinhole model, is that we think the distortion is not equal / circular all the way around the lens. So there may need to be some distortion parameters which are a function of the polar angle around the principle point.

If you dig up some lens distortion charts from reviews online you'll find that cheaper lenses with fewer elements have a bad tendency towards developing a sine-like curve starting somewhere maybe halfway out to the corners and going through a full cycle (so the chart looks roughly like a ripple spreading from the center). I'm not an optical engineer, but as far as I can tell this is impossible to correct without some minimal number of lens elements and corresponding increase in lens size barring technology advances since there's always a limit on what can be recovered from sensor data.

This paper and most other sources seem to call this mustache distortion and attribute it to variable focal length lens designs but I've seen it more often on cheap fisheyes (or fisheye adapters). It can be seen in a compressed fashion in circular fisheye images where the picture is blurred and bulged terribly towards then middle and begins to pinch together more and more along the outer radius.

Decentering tends to show up along with that particular issue (or even just with barrel distortion) especially on compact / budget APS-C fisheye / UW lenses where the lens might have a decent optical design but the company producing it is doing so on rejected parts or doesn't have the equipment to properly align internal elements.

We produce a 360 degree 3x12mp global shutter fisheye rig, and are keen to improve the quality of the meshes it produces.

If you're producing a product and not manufacturing lenses yourself (which I'm guessing you aren't), the manufacturer should be providing a distortion chart and some sort of guarantee on maximum deviation from it, or you're going to be stuck calibrating every single device sold if if you want them working right out of the box. If it's just a mustache or barrel distortion issue that affects every lens about the same you might be ok creating a profile that ships with it, but decentering issues in particular can produce wildly different centers for the barrel distortion between lenses and there's no recovering any of the data that was outside of the sensor area so you either have to detect and crop it somehow or do something more complicated.

That's if I understand what you're wanting to do correctly. If not, just ignore me. :-) That paper might help, and if you google it NVidia V-Ray has a standalone tool that will measure lens distortion (to apply to 3D scenes as lens shaders so they'll match shot footage when compositing, but it can be used to correct as well I think). I think Rawtherapee has downloadable charts and a profile generation tool as well but that isn't an API you can hook into easily.