Calibration file won't load

[marluca]

I generated a test calibration file with the stereoCameraCalibrator app. The mat file is attached to this issue. calib_test.zip

I tried to load this file into APT by clicking Label -> Select calibration file .. (There is no Load calibration file) but the following error occurs:

_Error using CalRig.loadCreateCalRigObjFromFile (line 145) Calibration file 'D:\Repositories\APT\calib_test.mat' has unrecognized contents.

Error in LabelerGUI>menu_setup_load_calibration_file_Callback (line 3181) crObj = CalRig.loadCreateCalRigObjFromFile(fname);

Error in gui_mainfcn (line 95) feval(varargin{:});

Error in LabelerGUI (line 48) gui_mainfcn(gui_State, varargin{:});

Error in matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)LabelerGUI('menu_setup_load_calibration_file_Callback',hObject,eventdata,guidata(hObject))

Error while evaluating Menu Callback._

I'd really appreciate help solving this issue.

Thanks!

[allenleetc]

Hi Marluca!

Yes I was able to reproduce your issue. When running your stereo calibration in the MATLAB Calibrator App, did you save/generate your MAT-file using the "Export Camera Parameters" button in the tool ribbon/strip?

Could you please try again but with saving your calibration using the "Save Session" button on the left of the tool ribbon instead? This produces a slightly different format and our code is written against these contents.

When you first load the calibration, some diagnostics will be run that internally calibrate some heuristics. This takes some time and creates some output so don't be alarmed. Some of this is opaque but some may be interesting (reprojection errors and the like).

[marluca]

Hi Allenleetc,

Thanks for your quick reply. Yes, exactly! Instead of saving the calibration via "Export Camera Parameters", I used now the "Save Session" button and tried selecting it as the calibration file for APT, but I got the following error:

Unrecognized method, property, or field 'BoardSet' for class 'vision.internal.calibration.tool.Session'.

Error in CalRigMLStro/autoCalibrateProj2NormFuncTol (line 131) bs = cs.BoardSet;

Error in CalRigMLStro (line 84) obj.autoCalibrateProj2NormFuncTol();

Error in CalRig.loadCreateCalRigObjFromFile (line 134) obj = CalRigMLStro(s.(vars{1})); % will auto-calibrate and offer save

Error in LabelerGUI>menu_setup_load_calibration_file_Callback (line 3181) crObj = CalRig.loadCreateCalRigObjFromFile(fname);

Error in gui_mainfcn (line 95) feval(varargin{:});

Error in LabelerGUI (line 48) gui_mainfcn(gui_State, varargin{:});

Error in matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)LabelerGUI('menu_setup_load_calibration_file_Callback',hObject,eventdata,guidata(hObject))

Error while evaluating Menu Callback.

I attached the new file as well. calibrationSession.zip

The original data used for the calibration can be downloaded here: http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html

Do these heuristics also affect the epipolar lines shown for labeling, or is it mostly used during or after training process?

[allenleetc]

Ah yes I see. Are you using MATLAB 2021a or later?

It looks like there have been some API changes in the Computer Vision toolbox. Should be an easy fix but am seeing something interesting with the Bouget sample calibration data. Will push the fix soon. The heuristics are definitely used for the epipolar lines and that is what is acting a bit strange with this dataset.

[marluca]

The Matlab version is '9.11.0.1769968 (R2021b)'.

Hm okey, I thought mostly the cameras' relative positions are necessary like the essential matrix. What else is needed except intrinsics/extrinsics/relative positions? Because I was thinking of using third-party calibration files and merging them into this format.

Thanks!

[allenleetc]

Added further commit fec3d0e588caa876a84f015c37cbcb83aee4dddd

@marluca I pushed some updates, can you please pull the latest (on the develop branch) and try your workflow again? Your calibrationSession now seems to be working for me.

A couple issues were at play. The first as mentioned was that some MATLAB/vision classes changed in 2021 (in a minor way). I updated our code to react and in the future we may insulate ourselves better from these classes.

The second issue was that I updated our epipolar line drawing implementation to rely more on MATLAB's Computer Vision Toolbox API. This seems to work better with the MATLAB-generated calibration on this Bouget sample data.

That said the MATLAB CV/calibration API is a bit rough and I am going to open a ticket with them as there are some oddities. Please let us know how things go and we can keep iterating if there are further hiccups.

[allenleetc]

Hm okey, I thought mostly the cameras' relative positions are necessary like the essential matrix. What else is needed except intrinsics/extrinsics/relative positions? Because I was thinking of using third-party calibration files and merging them into this format.

You are right, basically it is just the camera intrinsic/extrinsic parameters that are needed. Our current implementation happens to also use the extrinsic positions of the particular calibration board patterns as a starting point for some of the heuristic auto-calibrations I mentioned. When drawing the epipolar line, we currently sample 3d points along each epipolar ray, so as to fully incorporate nonlinear distortions. (This can lead to curved epipolar lines, unlike a linear closed-form solution that doesn't incorporate distortions.)

In order to sample the points semi-intelligently, we try to guess a range and spacing of z-values etc. We use the calibration board patterns as a crutch for this purpose. I'm guessing it would not be hard to remove this dependence.

If you don't mind sharing, what other/3rd-party calibration tools are you planning to use? This is an interesting topic.

[marluca]

Sorry, I just got back to it now.

Perfect, loading the calibration session works out with this commit!

So, I was using my own calibration pipeline that uses ChArUco boards in OpenCV, because of partial visibility and the like. Something that is quite similiar, and I might use for calibration only, is the following library: https://anipose.readthedocs.io/en/latest/start3d.html

In the end, you also get checkerboard corners from ChArUco boards but these are regressed from fiducial markers, which means it should also be rather easy to incorporate into the calibrate session format (as long as estimation errors are not really necessary). If I understand correctly, 3d board detections (and their associated 2d image points) are then needed in addition, right?

I think in general it would be nice to have an advanced option for not auto-calibrating, in case one can only access the extrinsincs, intrinsics and distortion coefficients like with anipose or some other software.

Wouldn't you as well get curved lines when using f.ex. the radial distortion parameters plus rectification as well? Which is probably not as sophisticated/stable as your technique. So, is there a paper or resource that explains the non-linear sampling approach/mapping? I'd be interested.

Thank you very much!

[allenleetc]

Great glad to hear you can get running! Thanks for the notes on your pipeline, makes sense.

I see your point about wanting an option to not auto-calibrate, when specific calibration board positions/detections are not available. Good suggestion and I think not difficult to implement.

Rather than requiring you to put your parameters into the MATLAB data structure, another option could be for APT to accept those parameters directly in some generic format. It's possible for instance that the relevant MATLAB objects are read-only or difficult to manually manipulate.

You are right that in general just the distortion parameters will result in curved epipolar lines. What we are currently doing is just that and nothing more sophisticated. We start in Camera1, undistort, extend a 3D ray out in Camera 1's positive z-direction, transform to Camera2, and then project/distort onto Camera2's image. The auto-calibration business is nothing fancy and just tries to numerically select a z-range and z-spacing for points on the epipolar line, rather than doing something geometric.

Let us know what you think, thanks!

[marluca]

Your suggestion is even better - it makes more sense to read these parameters directly into APT, also because there won't be API problems with the Vision toolbox again. And yes, too, bypassing the read-only Matlab objects is a mess.

So, I'd be definitely using this option and be glad if it could be implemented. I just got some recordings done this week, that I could use it for.

Thanks a bunch! Got it :)

P.S: A description or link to your calibration matrices format would then be helpful (if matrices need to be transposed like in Matlab f.ex.).

[allenleetc]

Cool, sounds good! Just FYI I am out on a trip for a bit but will put this on the todo list for when I get back.

[marluca]

I hope you're trip is going all well.

A bit related to this problem is the question on how to load calibration info via the calibrationSession for n cameras (n>2)? I tried a bit, but manipulating Matlab's CameraParameters attribute is restricted w.r.t. array formations. Which also makes sense because the StereoCalibrator is only usable for stereo calibration I suppose.

In the wiki it's mentioned that 3 cameras were used already; so do you have a rather fast solution for my case in mind? And, do you then draw multiple epipolar lines for more than 2 views or just propose the intersection?

Thanks!

[allenleetc]

Hey @marluca, thanks for your patience! I just got back and will be getting back to this soon.

For 3-camera support, do you have pairwise calibrations between all camera pairs (3 camera pairs in all)? If so, I think it should be straightforward to include 3-camera support for your case with this coming update. (We already have an N-camera calibration rig object that utilizes an array of pairwise calibrations between views.)

We were just discussing workflows for another 3-camera rig today. We would be curious about your thoughts here! As you note there are multiple possibilities, eg:

1. Show at most one epipolar line at a time (in any given view), eg for the most recently clicked/positioned landmark
2. Show/overlay multiple epipolar lines within views, eg for the last 2 or 3 clicked/positioned landmarks
3. If landmarks are clicked in two views, don't show epipolar lines, but instead show a reconstructed point in the third view
4. ... other combinations, etc ...

Way back when we had tried 3, but with a newer rig are thinking 1 or 2 might be preferable. Maybe the optimal solution depends on the rig/data/calibration. If you have any thoughts please send them along!

[marluca]

Hi @allenleetc ,

So, in my case I have probably 6 cameras available. So, it'd be definitely nice to interface with the N-camera calibration rig object. Yes, sure, I can compute the pairwise camera transformations (15 in total, I suppose). In a future update you might consider to include this step then into APT, because probably external calibration software won't calculate this. So that for each camera only R,t,K, and d are basically necessary.

---

I'm glad you ask! I think this might work out pretty well:

Procedure: Click a marker or several markers in one view. Go to another view. When clicking a marker, the corresponding epipolar line shows up. Reposition the marker. (Continue with other markers for that view if you like, where only one epipolar line would show up at a time). Go to another view. The marker or markers is/are repositioned by the intersection of the corresponding epipolar lines already (indicated visually by a r or sth. for reconstructed). Click at the/a reconstructed marker and only then the two epipolar lines will show up. Reposition it (if needed) and lock the location.

Reasoning: So for this procedure, you need to keep track if in one or two views the marker was already manually positioned. By doing so, you would allow per view and per marker labeling! Especially the latter becomes important in N-views, where there might be a lot of occlusion and this is, I think, one of the reasons for having N-views in the first place. Furthermore, the initialization saves a lot of time in the labeling process and is geometrically consistent, where humans might not be (especially for occluded markers). Also, keeping track in the labels of which reconstructed points were not relocated but simply accepted makes sense. Showing epipolar line/s only when clicking on a marker prevents the image being too crowded.

---

Thoughts regarding the optimal solution: In general I think it's possible to select epipolar lines by how well cameras are calibrated w.r.t. each other, but this selection only happens if there are more than two views with a visible marker and make your software then again a bit more specific as this needs to be provided somehow by the calibration.

[allenleetc]

Hey those are some great ideas! Having the EP lines show up only when/after a point is clicked/selected definitely seems interesting. @iskwak

Okay first things let's get this simplified initialization in.

[allenleetc]

@marluca just FYI I checked a prototype solution into https://github.com/kristinbranson/APT/commit/d5257ff272f372535369449f9db195c054261867 (branch iss375).

There is an example camera/rig specification file stereo_cam_rig_example.yaml. The parameters are taken from your example calibration! This format is still preliminary.

If you want to try this out, pull the above commit/branch, open your two-view project, and then under Label>Select Calibration File, select the rig specification yaml file above. This will create and initialize a CalRig object using stereo calibration parameters in the yaml.

Some notes:

• This was working for me but is not fully tested yet!
• Currently 'fisheye' calibrations are not supported. Are you using those?
• When loading calibrations from the yaml, the auto-calibration business is not done. However the epipolar line drawing code has not been updated. So, you may note that the epipolar lines do not extend fully through the views.

Any/all thoughts/suggestions welcome, let us know!

[marluca]

@allenleetc Yes, perfect, I was able to load the 2-camera rig calibration in. I proceeded with loading my 'real' calibration for 2 cameras. I noticed then two things:

1. The Rotation matrix, which is given as a Rodrigues vector, was constructed from the transposed Rotation matrix? (consistent with Matlab's row-vector notation; that's also the output of Matlabs StereoCalibrator)
2. The epipolar lines as you said are not extending through the views fully. In fact, I have to zoom out on one image really far to have it shown on the other.

Apart from that:

• I do use mostly 3 radial distortion parameters so far.
• Is it possible to maintain this format, and extend it to N cameras, so the pair-wise calibration is done within APT/Matlab (as for 2 views)? I think that would be really convenient.

Thanks a lot, looking forward to new updates!

[allenleetc]

@marluca Great thanks for trying that!

Re: the Rotation Matrix. I actually thought my definition was transposed relative to MATLAB; their row notation seems less idiomatic to me. To clarify, in APT the definition of R and t are supposed to be as in the following. [x;y;z] is a vertical column vector, and same for [p;q;r].

# - Rotation (R), Translation (t) defined so that
#
#     [x;y;z] = R*[p;q;r] + t,
#
#       where (x,y,z) and (p,q,r) are in the camera/"world" coord system respectively

Did I get this wrong or flipped?

Re the radial distortion coefficients, sure we can add/accept a third coefficient. For N cameras, yes should be easy to compute the pairwise extrinsic transforms between cameras to initialize a "pairwise-calibrated" rig object (CalRig). Will be continuing these updates and integrating into the develop branch.

[marluca]

@allenleetc No, the definition and your format is alright. I just assumed the code uses still the transposed one, because you used the Matlab Bouget sample calibration data and my calibration data needed to be transposed to start drawing epipolar lines (although I had to zoom out a lot). But of course you transformed the rotation matrices into a vector, so did you transpose prior to that? Cheers

[allenleetc]

@marluca Ah yes I did transpose the "RotationOfCamera2" matrix from your MATLAB calibration session before transforming into a rodrigues vector.

Will be getting back to this. Side note FYI. If you want to manually set the "z-range" of your epipolar lines, you can manually specify the range as follows after loading your project and setting the calibration file:

lObj = StartAPT;

% load project, set/select calibration file/yaml. I applied the calibration to all movies

calrig = lObj.viewCalibrationData; % scalar 'CalRigMLStro' object 

% increases the z-depth used (relative to default) when generating EP lines from clicks in view1/camera1
calrig.eplineZrange{1} = 0:.1:300; 

% click in view1; the EP line should now extend further into view2

[allenleetc]

Hey @marluca, pushed a fix for 'incomplete' epipolar lines. Now using a geometric procedure by default; no z-sampling or calibration pattern locations required. I also added the 3rd radial distortion parameter to the stereo rig yaml file.

This is all integrated into the develop branch, let me know if this works for you! Note that if you have an existing project, it's probably easiest to reload/re-specify the calibration for your project (via Label > Selection Calibration File) if you want to add in a 3rd radial distortion coefficient.

Minor tech note: the previous z-projection method for drawing epipolar lines had the side benefit that unphysical/negative-z points were not drawn. With your example stereo pair (cameras side-by-side), the geometric approach draws a full epipolar line which includes points "past infinity" along positive z. Kind of a fun note.

For the N-camera rig, if you have an example rig yaml (eg for 3 cameras) please send along as I can use it for testing.

[marluca]

Hey @allenleetc , sorry for my late response here. Thanks a lot for the new epipolar line drawing, it works now! Great! The geometric procedure fixed it rather than the transposing, I suppose.

Ye, the negative z-values are a bit unfortunate, but should only occur if the other camera is visible in the frame, right?

Sure, I attached a 3-camera rig below. calib.zip

[allenleetc]

Sweet thanks for the example rig!

Re: negative z-values, it would certainly occur in the example you describe. But it occurs in other cases as well. For instance suppose you have two cameras aligned side by side (call them L and R), with their positive z/optical axes parallel extending to infinity.

Assume the principal points of each camera are in the centered in their images. Consider the principal point in L and its epipolar line as projected in R. If we consider only positive z-values, this line will "come from the left" in R and then never go past the middle of the R image (it will asymptotically approach the principal point of R). However, the usual definitions I've seen define the epipolar line as the entire line, including points in the right half of R.

In this example I believe those points will be mathematically generated if by projecting the point in L in the negative-z direction.

Hope that makes sense? Anyway it's mostly just a curiosity I imagine but kind of fun. Maybe it could even be relevant for labeling on the rare occasion.

[marluca]

Ah, yes. You're right, thanks! Unfortunately, I haven't seen more restrictive definitions either.

I suppose labeling points beyond 'infinity' would mess up the triangulation. So this might be dealt with in the future.