Open willtoth opened 6 months ago
mcm001
commented
6 months ago I've seen this once before but attributed it to problems converting from photon to mrcal lensmodel file, not an underlying difference. I think the next thing to do is set breakpoints in mrcal and compare seed poses and detected corners and stuff?
willtoth
commented
6 months ago The first time I ran this, the intrinsics calculated between the two were still different, but quite close to eachother. e.g.
mrcal
[ 901.7330686, 902.7952017, 799.8291256, 666.452847, 0.05673155155, -0.08460947181, -0.0009418336749, -0.0008824814115, 0.01479466453, 0.003773460448, 0.006533366048, -0.01477754318,]PV
[ 900.3160751, 901.470041, 799.8930541, 667.18491, 0.05164757998, -0.08876075177, -0.0006819796758, -0.0008676279577, 0.02579295175, -0.0003870303465, 0.003100190011, -0.004102380634,]The camera and lens itself is the same model between this and the one in the original message. The values are pretty close. Either way, for the intrinsics calculation, do you expect the mrcal CLI or the results from the PV run to be more accurate?
I won't have time to help debug for the next 2 or so weeks as we prepare for our first event, but I can probably dig into it a bit after that.
randomstring
commented
6 months ago I too am seeing differences between the PV output and what mrcal computes. When I map the uncertainty of the PV calibration I get an all yellow graph like the one linked above.
running calibration like this: (8x8 grid with 1" spacing, correct me if I'm getting the arguments wrong.)
mrcal-calibrate-cameras --corners-cache corners.vnl --lensmodel LENSMODEL_OPENCV8 --focal 1900 --object-spacing .02540125 --object-width-n 7 '*.png'I get a more reasonable map of error, with a sweet spot in the middle in the 0.2 error rage with more error around the edges.
mcm001
commented
6 months ago For continuity , are the intrinsics you get different? Or is only the graph.
randomstring
commented
6 months ago Yes, intrinsics were slightly different, mostly values just a few percent different: 685.66 -> 686.27, etc.
dkogan
commented
6 months ago Hello. I'm the mrcal author. Feel free to ask me for help about things mrcal itself is doing; use the mrcal bug tracker maybe. For photonvision problems, I cannot say much, but I'm happy to help @mcm001 or whoever else debug stuff.
mcm001
commented
6 months ago I think that debugging this is probably going to require I sit down with a debugger and compare inputs to mrcal line-by-line against your implementation again.
Does mrcal-calibrate-cameras by default perform a calibration refinement? Right now, our calibration is just a single call to mrcal_optimize (see here). And we generate seed poses using solvepnp_epnp.
mcm001
commented
6 months ago camera-0_RESOLVE_WITH_MRCAL_CALIBRATE_CAMERAS.zip
Here's the cameramodel I got from using mrcal-calibrate-cameras, which again matches the results you got using the mrcal CLI. I then wrote a little unit test to use your calibration JSON (attached in your comment above) and try calibrating. This is the output I see.
@Test
public void testWillBoard() throws IOException {
var cal = JacksonUtils.deserialize(Path.of("/home/matt/mrcal_debug_tmp/pv_calibration_1600x1200.json"),
CameraCalibrationCoefficients.class);
// System.out.println(cal);
var corners = cal.observations.stream().map(it->{
var mat = new MatOfPoint2f();
mat.fromList(it.locationInImageSpace);
return mat;
}).collect(Collectors.toList());
var result = MrCalJNI.calibrateCamera(
corners,
(int)cal.calobjectSize.width, (int)cal.calobjectSize.height, cal.calobjectSpacing,
(int)cal.resolution.width, (int)cal.resolution.height, 1200
);
System.out.println(result);
}Calibrate3dPipeTest > testWillBoard() STANDARD_OUT
CameraCalibrationCoefficients [resolution=1600x1200, cameraIntrinsics=JsonMat [rows=3, cols=3, type=6, data=[964.3192542051779, 0.0, 766.3572370964357, 0.0, 968.4167197906467, 656.0259427916941, 0.0, 0.0, 1.0], wrappedMat=Mat [ 3*3*CV_64FC1, isCont=true, isSubmat=false, nativeObj=0x7fe345180730, dataAddr=0x7fe345180f40 ], wpilibMat=null, wrappedMatOfDouble=null], distCoeffs=JsonMat [rows=1, cols=8, type=6, data=[-0.03374426143309665, 3.779715023416348E-4, 8.241449288523903E-4, -0.009869350879493375, -1.3818166014360312E-6, -0.050844745308818114, 7.160772275740329E-4, -3.0046612000201336E-6], wrappedMat=Mat [ 1*8*CV_64FC1, isCont=true, isSubmat=false, nativeObj=0x7fe34510dc80, dataAddr=0x7fe34519b680 ], wpilibMat=null, wrappedMatOfDouble=Mat [ 1*8*CV_64FC1, isCont=true, isSubmat=false, nativeObj=0x7fe34510dc10, dataAddr=0x7fe3451ab080 ]], observationslen=80, calobjectWarp=[-2.25221267024838E-4, 7.336652791935415E-4], intrinsicsArr=[964.3192542051779, 0.0, 766.3572370964357, 0.0, 968.4167197906467, 656.0259427916941, 0.0, 0.0, 1.0], distCoeffsArr=[-0.03374426143309665, 3.779715023416348E-4, 8.241449288523903E-4, -0.009869350879493375, -1.3818166014360312E-6]]
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: CHOLMOD warning:
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: not positive definite.
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: file: ../Cholesky/t_cholmod_rowfac.c
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: line: 430
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114:
Watching 173 directories to track changes
/home/runner/work/mrcal-java/mrcal-java/mrcal/mrcal.c(5540): Threw out some outliers. New count = 425/37440 (1.1%). Going again
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: CHOLMOD warning:
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: not positive definite.
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: file: ../Cholesky/t_cholmod_rowfac.c
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114: line: 430
libdogleg at /home/runner/work/mrcal-java/mrcal-java/libdogleg/dogleg.c:1114:
Gradle Test Executor 13 finished executing tests.
> Task :photon-core:testHeadless
MrCalResult [success=true, intrinsics=[815.2444290805902, 702.5299171031456, 783.7045709624529, 598.5380702840318, 16.022877352468576, -11.617096993914174, -0.012759261238650944, -0.16064034039996528, -6.856871701238218, 11.279315510816447, -8.68659166855614, -4.641638738271045], rms_error=48.3446650298785, warp_x=0.005452025977464822, warp_y=0.0011008387439970853, Noutliers=425]Does mrcal-calibrate-cameras by default perform a calibration refinement? Right now, our calibration is just a single call to mrcal_optimize. And we generate seed poses using solvepnp_epnp.
mrcal-calibrate-cameras does this also: there's an incremental seeing and solving routine. It's implemented in Python, which I guess means you can't use it directly. The seed routine is here:
Look at mrcal-calibrate-cameras to see how that's called.
dkogan
commented
5 months ago I looked at the model in the .zip file and the model in the smaller .json file above (mrcal models aren't json: they're python dicts; so they can have comments, and such). Both those models show the "small" uncertainties.
Is this bug report summarized as "photonvision produces results with alarmingly-high reported uncertainties"? If so, I'd need a model that shows such high uncertainties to tell you what's going on. Are the high-uncertainty produced by running mrcal-show-projection-uncertainties, or is that wrapper inside photonvision somehow? I'd need some way to reproduce whatever failure you're seeing.
Similarly for the "not-positive-definite" messages you're seeing: if I reoptimize either for the models I have, I don't see this. The most common cause of these messages is described here:
https://mrcal.secretsauce.net/how-to-calibrate.html#orgea10dd0
mcm001
commented
5 months ago Yeah, that summery is accurate. For a little more context, Photon uses JSON files for camera calibration, and these jsons contain the images, board corner observations, and other optimization inputs. We also provide a python script to convert this json to a mrcal cameramodel, and try to convert out optimization inputs to mrcal-land, so we can use your tools like mrcal-show-projection-uncertainty. The core bug report here is that when a calibration performed using PhotonVision is exported and converted to a mrcal cameramodel, the projection uncertainty shown using mrcal-show-projection-uncertainties is alarmingly high.
A bug could exist in our export/conversion script, or we could truly have a calibration that's wildly wrong.
mcm001
commented
5 months ago mrcal-calibrate-cameras does this also: there's an incremental seeing and solving routine. It's implemented in Python, which I guess means you can't use it directly. The seed routine is here: https://github.com/dkogan/mrcal/blob/90818fdbb6ef422c90b644690d5b14add3a42724/mrcal/calibration.py#L1360 Look at mrcal-calibrate-cameras to see how that's called.
right now we just jump straight into a full solve optimizing everything simultaniously, which probably doesn't help. I'll try changing our code to properly do the same incremental optimization process you linked and see how that changes our results.
mcm001
commented
5 months ago I've started prototyping this code over in this branch, but i'm still getting weird results. Needs more investigation
dkogan
commented
5 months ago The core bug report here is that when a calibration performed using PhotonVision is exported and converted to a mrcal cameramodel, the projection uncertainty shown using mrcal-show-projection-uncertainties is alarmingly high.
OK. Can I get a copy of one of those .cameramodel files that produce a very high mrcal-show-projection-uncertainty result? I believe all the ones that were linked in this report have reported low uncertainties
right now we just jump straight into a full solve optimizing everything simultaniously, which probably doesn't help. I'll try changing our code to properly do the same incremental optimization process you linked and see how that changes our results.
I won't claim that the seeding sequence I use currently is perfect, but it has been tested thoroughly, and does work ok. Is there really no way to use the Python code directly in photovision? If doing a rewrite in another language, it'll take a while (lots of testing and fixing things) to get to the same level of performance, and it feels like effort is being duplicated. You can embed Python in C like this:
https://docs.python.org/3/extending/embedding.html
Worth thinking about?
mcm001
commented
5 months ago The main reason I'm hesitant to pursue that is it would require we install python3 and mrcal in our image. This would add ~450MB of packages to our image (at least for the mrcal package, version 2.4.1-1jammy1). Just python3 itself adds ~25MB, so I assume most of this extra size is from other dependencies the mrcal package pulls in?
If there's any way to get the size of the package down while still being able to wrap python, that could be something we consider for 2025. You're right that I'm not a fan of how much API I'm having to duplicate. I'd just also like to be cognizant of image size.
dkogan
commented
5 months ago You're talking about the debian packages here, and the "450MB" number comes from what APT tells you in will need when you ask to "apt install python3-mrcal"? If not, where does "450MB" come from? Presumably you're already installing "libmrcal4" and "libmrcal-dev", and you're looking at the EXTRA space "python3-mrcal" will require?
If so, can you get some more detailed accounting, to see which packages specifically are responsible for the ballooning size? mrcal itself and things it directly uses aren't nearly this fat, but there could be optional things, or things that dependencies optionally need.
I can help you get the detailed numbers, if you tell me exactly where you get the "450MB" number.
mcm001
commented
5 months ago Sure. Starting from a blank ubuntu:22.04 docker image, I'm just using the size reported by apt (eg, After this operation, 264 MB of additional disk space will be used.). After i list the extra size of the package, I go ahead and install it.
apt-get install --yes openjdk-17-jre-headless -- 260 MB
apt-get install --yes libcholmod3 liblapack3 libsuitesparseconfig5 -- 14 MB. These packages are what mrcal-java links to. Mrcal-java is a build that contains both mrcal and our JNI in the same so. mrcal-java shared libraries live inside of the photonvision JAR file we distribute, and on linux that's 107kb compressed in a jar.
libmrcal4 -- 24 MB. I don't think it makes sense to install libmrcal-dev on user systems right, since they don't need to be compiling code?
python3-mrcal: after installing libmrcal4, an additional 1615 MB
mcm001
commented
5 months ago sorry, github formatting was messed up initially for my above comment. should be resolved now.
dkogan
commented
5 months ago OK. I just poked at it a bit. If I ask for the size of everything needed to install python3-mrcal that isn't also needed for libmrcal4, it says there's 1159MB extra:
for pkg (`join -v1 <(apt-rdepends python3-mrcal | grep -v '^ ' | sort) <(apt-rdepends libmrcal4 | grep -v '^ ' | sort)`) { echo $pkg/bookworm } | xargs apt-cache show | awk '/^Package:/ {p=$2} /^Installed-Size:/ { s+=$2 } END {print s/1e3}'It's a bit different from your other numbers for uninteresting reasons; hopefully much of the reasoning would be the same either way. These are installed sizes. I looked at it a bit, and it looks like 735MB of that is python3-opencv. This package definitely isn't as fine-grained as it could be. So today, if you want to be able to import cv2 in Python, you need 735MB of space.
Of the remaining stuff, I THINK much of it is from installing the gnuplot-qt flavor of gnuplot, which pulls in a lot of stuff. There are other flavors of that, which would be lighter. So what do you see if you
apt install --no-install-recommends python3-mrcal gnuplot-x11Hopefully that will give you a better number. Although the biggest hog (python3-opencv) will still be there.
I'm still not sure why we care about this though. Disks today are very large (which is why these packages are fatter than you want: nobody has cared enough to trim them). You are using a very common OS with very common packages. Why do you even need an image? You can use something like equivs to build a (tiny) meta-package that depends on the things that you need (let's call it photon-vision-deps). Then the instructions are:
photon-vision-depsThen even if photon-vision-deps contains some fat thing like python3-opencv, it isn't something you think about. And the students won't either: it's a fraction the size of a netflix movie.
And as an added bonus, having Python and opencv available is something the students should find useful.
Sorry if that's not completely what you wanted to hear. If you had infinite time and motivation, you could improve the python3-opencv packaging to make it more granular, to allow installing only the pieces we need. But that's not obviously a good use of anybody's time, which is why nobody has done it.
If you need help with equivs or setting up an APT server with the extra packages you need or whatever, feel free to ask. I do this all the time.
mcm001
commented
5 months ago Robots are generally an offline environment, which drives the need for a completely offline solution. For competition, we also need to disable wireless, which forces you to somehow find a wired internet port at your school/workshop. I agree that in a perfect world, you'd just plug your coprocessor into your school LAN and download some packages, but that's outside the reach of a good chunk of our users.
The concern about total disk space is to keep image download/flashing speed from bloating too much, and also just proactively trying to not hit Github's download size limit. I'm just not sure if the value prop is there for increasing the size of our image this much is there yet.
From a fresh docker image of ubuntu:22.04, I see:
dkogan
commented
5 months ago Robots are generally an offline environment, which drives the need for a completely offline solution. For competition, we also need to disable wireless, which forces you to somehow find a wired internet port at your school/workshop. I agree that in a perfect world, you'd just plug your coprocessor into your school LAN and download some packages, but that's outside the reach of a good chunk of our users.
I don't follow. Currently you download an image from the internet and use it to initialize an OS with some packages; then you can be offline. How's that different from downloading some packages from the internet and then going offline?
The concern about total disk space is to keep image download/flashing speed from bloating too much, and also just proactively trying to not hit Github's download size limit. I'm just not sure if the value prop is there for increasing the size of our image this much is there yet.
I don't know your system at all, so I'm going to stop proposing things. If you want to have a separate conversation about Debian packages and apt servers and managing installs and whatnot, we can do that separately from this issue. I'm a Debian Developer, and have managed many such things previously.
From a fresh docker image of ubuntu:22.04, I see:
... After this operation, 1497 MB of additional disk space will be used.
OK, so we saved ~ 100MB by picking a different gnuplot? That's not nothing, but it's also not everything either.
mcm001
commented
5 months ago Yeah, that's something! When we're talking about the co-processors that people run photon on, they tend to be completely headless setups -- is it worth trying to avoid installing gnuplot/other GUI tools altogether?
Photonvision also ships with opencv (not installed as a package, but as just shared libraries from our opencv fork here embedded in the JAR, so it seems duplicative to also install opencv packages assuming mrcal could work with opencv 4.8.0. And you only need a couple modules from opencv for a vanilla monocular camera calibration right? (I saw solvePNP used, but I think that was it). I see used:
I also noticed that python3-opencv is using packages named libopencv-[MODULE]4.5d -- does this mean that they're packages with debug symbols?
I think if we can get opencv trimmed down a bit, this could be a viable option to pursue once the season wraps up
mcm001
commented
5 months ago Another thing I'll throw onto the pile for consideration. Our current JNI setup doesn't require Windows users install any additional programs to make Photon work. If we go the path of directly calling mrcal-python, that won't be true anymore. I think that whatever solution we end up at, making sure it work painlessly under windows is a hard requirement.
dkogan
commented
5 months ago Matt @.***> writes:
they tend to be completely headless setups -- is it worth trying to avoid installing gnuplot/other GUI tools altogether?
Some things are still useful: you can x-forward gnuplot for instance. But the gnuplot packages have flavors of different fatness (because those packages were built when disk space was more at a premium). You certainly don't need most of opencv, but it would take some work to disentangle that
Photonvision also ships with opencv (not installed as a package, but as just shared libraries from our opencv fork here embedded in the JAR, so it seems duplicative to also install opencv packages assuming mrcal could work with opencv 4.8.0.
That's on photonvision for doing something weird. It should be installing and using the system packages.
And you only need a couple modules from opencv for a vanilla monocular camera calibration right? (I saw solvePNP used, but I think that was it). I see used:
- cv2.findContours
- cv2.contourArea
- cv2.solvePNP
That sounds right.
I also noticed that python3-opencv is using packages named libopencv-[MODULE]4.5d -- does this mean that they're packages with debug symbols?
The meaning of the "d" is a question for whoever wrote the opencv build system. The debian packaging strips out the debug symbols into separate packages (called whatever-dbgsym). All the package sizes we've been looking at do not include the debug symbols.
I think if we can get opencv trimmed down a bit, this could be a viable option to pursue once the season wraps up
Let's talk separately from this issue later, when you actually want to undertake bigger surgery.
mcm001
commented
5 months ago // Fill in object/image points
for (int i = 0; i < boardSize.height; i++) {
for (int j = 0; j < boardSize.width; j++) {
- auto &corner = c_observations_board_pool[i * boardSize.height + j];
+ auto &corner = c_observations_board_pool[i * boardSize.width + j];Found the bug! Bad iteration bounds in where we fill in our chessboard corner array. I found it by just comparing inputs to cv::solvePNP between mrcal-calibrate-cameras and our JNI. This was causing solvepnp to give totally wrong seed poses, leading to the not positive definite warnings (I think). I now get basically identical outputs from the two calibration APIs:
cpp: Intrinsics [12]: 895.605941 896.407818 829.075501 665.325699 -0.001086 -0.007815 0.000080 0.001626 0.116579 -0.049334 0.064817 0.100289
mrcal-calibrate-cameras: 895.5869583, 896.3883938, 829.0801411, 665.3252287, 0.004235123928, -0.01917739877, 7.928632284e-05, 0.001627242293, 0.107986625, -0.04425876899, 0.05428592994, 0.09100042495Awesome! Glad you found that. Let me know if you hit any other issues that I can help with.
We ran calibration with mrcal enabled on ~80 images. At the end of the process, the intrinsics output were
Running the same images using calibrationUtil.py to extract them, but with the mrcal CLI as described here, the intrinsics result is:
Some data in the mrcal CLI output
I also recall the output (number of outlines) being much higher on the PV UI, but I did not capture this unfortunately.
Additionally, the calibration results by running
mrcal-show-projection-uncertainty --unset key ./cameramodel_0.cameramodelis quite poor with the output of the python script, but shows quite good values with the output of the mrcal.Values generated from calibrationUtils.py
mrcal CLI
Platform (PV):
Platform (mrcal CLI):
JSON file is too big for Github, download here
mrcal results mrcal_camera-0.cameramodel.json