Mrcal C++ test script `ASSERTION FAILED: ctx->factorization != NULL is not true`

[mcm001]

Hello! I'm interested in integrating mrcal with PhotonVision, a Java-based computer vision project for FIRST Robotics teams; I've been super impressed by the Python camera calibration scripts so far, and have had much better results vs the stock opencv camera calibration. We're open to just directly wrapping the CLI, but would prefer to interact via JNI if it's low enough friction for us to implement. I see there's a chunk of work that happens in mrcal-calibrate-cameras which we'd loose out on, so open to suggestions/opinions.

Right now I'm using libmrcal-dev/jammy,now 2.3-1jammy1 amd64 and mrcal/jammy,now 2.3-1jammy1 amd64. I took some of our (rather poor) calibration test images I had floating around, and using the mrcal-calibrate-cameras script, got a calibration output that looks sane:

``` matt@photonvision ~/D/G/mrcal-java (main)> cd /home/matt/Documents/GitHub/mrcal-java ; /usr/bin/env /bin/python3 /home/matt/.vscode/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debug py/launcher 36071 -- /usr/bin/mrcal-calibrate-cameras --corners-cache /home/matt/Documents/GitHub/photonvision/test-resources/calibrationSquaresImg/piCam/640_480_1/corners.vnl --lensmodel LENSMODEL_OPENCV8 --focal 1200 -- object-spacing 0.0254 --object-width-n 7 /home/matt/Documents/GitHub/photonvision/test-resources/calibrationSquaresImg/piCam/640_480_1/\*.jpg ## initial solve: geometry only mrcal.c(5335): ## Nmeasurements=784, Nstate=48 # iteration step_accepted norm2x_before norm2x_after step_len_cauchy step_len_gauss_newton step_len_interpolated k_cauchy_to_gn step_len step_type step_direction_change_deg expected_improvement observed_improvement rho trustregion_before trustregion_after 0 1 4816.87 4815.84 5.35662e-05 0.00801267 - - 0.00801267 gaussnewton - 1.02801 1.02613 0.998164 1000 1000 1 1 4815.84 4815.84 1.495e-07 0.00134932 - - 0.00134932 gaussnewton 107.042 0.00349168 0.00332269 0.951604 1000 1000 2 1 4815.84 4815.84 1.66956e-08 0.000246365 - - 0.000246365 gaussnewton 111.987 0.000119432 0.0001093 0.91516 1000 1000 3 1 4815.84 4815.84 3.64414e-09 4.78132e-05 - - 4.78132e-05 gaussnewton 119.066 4.65922e-06 4.04768e-06 0.868746 1000 1000 4 1 4815.84 4815.84 7.89594e-10 9.86935e-06 - - 9.86935e-06 gaussnewton 127.466 2.15703e-07 1.76891e-07 0.820067 1000 1000 5 1 4815.84 4815.84 1.91059e-10 2.21809e-06 - - 2.21809e-06 gaussnewton 137.316 1.18058e-08 9.22137e-09 0.78109 1000 1000 6 1 4815.84 - 4.8823e-11 5.3388e-07 - - 5.3388e-07 gaussnewton 147.009 7.30162e-10 - - 1000 - ## RMS error: 2.478437986814644 ## initial solve: geometry and LENSMODEL_STEREOGRAPHIC core only =================== optimizing everything except board warp from seeded intrinsics mrcal.c(5414): Threw out some outliers. New count = 1/784 (0.1%). Going again ## final, full optimization ## RMS error: 0.3047116986020488 RMS reprojection error: 0.3 pixels Worst residual (by measurement): 1.2 pixels Noutliers: 1 out of 392 total points: 0.3% of the data calobject_warp = [-0.00148937 -0.00203832] Wrote ./camera-0.cameramodel ```

So far, I've copy-pasted some seed values from Python from a known working calibration into this test C++ script I put together. It's pretty hacky, I just wanted to quickly see how hard this would be. Currently, it spits out this error from somewhere deep inside libdogleg.

mrcal.c(5335): ## Nmeasurements=784, Nstate=16
# iteration step_accepted norm2x_before norm2x_after step_len_cauchy step_len_gauss_newton step_len_interpolated k_cauchy_to_gn step_len step_type step_direction_change_deg expected_improvement observed_improvement rho trustregion_before trustregion_after
libdogleg at dogleg.c:510: ASSERTION FAILED: ctx->factorization != NULL is not true
[1] + Done                       "/usr/bin/gdb" --interpreter=mi --tty=${DbgTerm} 0<"/tmp/Microsoft-MIEngine-In-sb4qhr1u.wc2" 1>"/tmp/Microsoft-MIEngine-Out-zjlfjru5.hko"

Do you see anything obviously wrong I'm missing that could cause this?

[mcm001]

Quick update, I did actually manage to solve that one! I had some weird options selected for mrcal_problem_selections_t, and I also was using a rather old version of mrcal. I built locally from HEAD, and fixed that, and I now get identical output to Python!

I'm still curious to hear thoughts you have on wrapping the CLI vs writing a JNI.

[dkogan]

Hello!

I'm glad people are trying out the library. So, to be clear, you were hitting this assertion using an old mrcal version (which one? what it a release?) but you updated to the master, and it works now? If it's not too difficult, can you try again using the v2.3 release tag? It'd like the 2.3 release to work, since the current master has LOT of changes, and is quite far from being releasable.

As for what is the most appropriate thing to be wrapping, that really depends on your use case. As you saw, the core mrcal functionality is implemented in the C library, but all the higher level stuff (that you probably want!) is written in Python. So if you wrap the C library, then you lose lots of functionality. Which may be just fine for your application. Also, note that the C API isn't stable, and will change somehwat with each release; at least that's the case now.

I hadn't considered interfacing with other languages at all, and would like to hear your thoughts on the matter. What would an ideal java-friendly mrcal look like?

[mcm001]

I'd installed these packages on my Ubuntu 22.04 WSL VM, if that's helpful: libmrcal-dev/jammy,now 2.3-1jammy1 amd64 and mrcal/jammy,now 2.3-1jammy1 amd64. I'll work on a MVP to reproduce the bug I was hitting above to send you, as I think that's probably the most helpful from my end.

And my little Java wrapper library is also working through Java now! What I've got implemented in my JNI here is basically the feature set of what I think we (as Photon) would need out of a java-friendly mrcal. I had to roll my own optimizer seeding, but it was largely copy-paste solvePNP code from your python code, but that was all I was missing on the "actually run calibration" part that was implemented in Python. Because we only need monocular calibration, we can do without a lot of the additional stereo complexity.

We do also loose out on the pretty killer graphing utilities you've got implemented in Python. But drawing quiver plots is something I have a pretty good handle on, and I'm happy to duplicate in our backend Java (or frontend JS?) code. We also have some unique constrains here since PhotonVision is designed to run on a headless coprocessor and serves a web-based UI/dashboard to clients. At the end of the day, for our use case the secret sauce is the optimizer, we can make our own graphs given data to plot.

And I personally don't mind some headache from the unstable internal C API. Us maintaining a JNI wrapping one function, that we upgrade a handful of times a year for the annual FRC competition season, is pretty low overhead imo. As for what the Right Way to do this is, I'm not super sure. I sure hope Java-based computer vision is a pretty niche field (Python or C++ are absolutely the better language choices for this problem-space), so I wouldn't devote too much time to Java support unless we aren't the only interested party.

[mcm001]

Forgot to attach this screenshot of mrcal-java too! It's still very much sort of MVP, but I was looking at like >1000ms with opencv's calibration to <20ms with mrcal in a rather poor test with 10 hastily taken snapshots of a small chessboard. This doesn't hold across my slightly better calibration datasets, but it's still a huge improvement. Hopefully this means our users won't have to wait literally minutes for calibration on Pi-3-and-comperable devices.

[dkogan]

Cool! Glad you got that running. This is going to be used in FRC competitions? You're making a simplified library that the students are going to use? Exciting.

And not to give you homework or anything, but if you're looking for a Java-native vision library, look at boofcv. I obviously think that my tools are better, but you might be able to make use of boofcv more directly since it's already in java.

[mcm001]

Yeah, exactly! PhotonVision is actually already an established open source vision software thing, we've been around for a few years at this point. FRC is moving towards using apriltags for robot localization (instead of retro reflective tape markers), but that means teams need to do a better job of camera calibration to get accurate robot localization, and our current opencv camera cal code was struggling especially on some embedded devices. (It's actually super exciting that high schoolers are getting exposed to these real hard robotics problems, it's been super useful to me professionally too)

Yeah I'll take a look at BoofCV! I had no idea anyone was crazy enough to implement all that in pure Java. From what I've seen I think mrcal is the better tool here, but I'll poke at it regardless.

And I've still got an MVP on the latest stable release of mrcal still on my todo list :)

[dkogan]

Matt @.***> writes:

And I've still got an MVP on the latest stable release of mrcal still on my todo list :)

Yeah. If we can chase down the crash, that would be useful. If you have an easy way for me to reproduce, I'm happy to debug it myself.

[mcm001]

Well I have good (or bad?) news. I took my working code from HEAD, and rolled back to 2.3-1jammy1 amd64 from apt, and was unable to reproduce. I think that my original issue can be attributed to an implementation error on my end. I think we're probably good to close this? But happy to keep you in the loop if we end up getting anywhere with mrcal in photonvision, if you're interested!

[dkogan]

Perfect. I'm happy to call this done. And I'm definitely interested in hearing about the progress. How soon are you hoping to have students actually using this thing? I'll be doing some minor mentoring to a local FRC team later, and it'd be cool if they actually end up using my tools. Thanks much

[mcm001]

Awesome to hear you're mentoring! Pending developer bandwidth (I'm busy with an internship this fall, other people are hitting midterm season at college and stuff), it would be awesome to have an MVP for Linux platforms before the season starts in January. I'm like 90% of the way to this already, but the last 10% is going to take another 90% of the time.

And if you have the time and inclination, we (photonvision) would love more developers or software testers! We've got a fairly active group on discord if that's your speed, would love to catch up there sometime.

[dkogan]

Hi. I really cannot devote any time to this, unfortunately. But definitely email me (or make an issue here) for any mrcal-related questions.