HoloLens 2 ir cam intrinsics

[mrcfschr]

I'm trying to track a surgical tool using the Hololens' IR sensor and PSI. Since the psi intrinsics are only estimates, I was wondering if someone in the community knows a better way to get more accurate intrinsics? As it is right now, we have a bit of an offset (a couple of mm) when tracking the tool. Or maybe I should just use the MapImagePointToCameraUnitPlane function directly instead of the intrinsic estimates? Would that help? I'm actually not entirely sure where the offset is coming from.

Thank you!

[sandrist]

Yes, maybe experiment with the MapImagePointToCameraUnitPlane function directly and see if that gives you better/different results? Beyond that, take a deeper look at the Research Mode APIs to see if there is anything there. We use Research Mode libraries to expose the IR sensor to \psi, but there might be more there we could leverage for computing intrinsics. Not sure though.

[Abdul-Mukit]

@mrcfschr hi, As far as I remember, I had performed camera calibration using a board. https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html

I captured a couple of images from the IR sensor. They were in 16 bits so I had to convert them to 8 bits. OpenCV's histogram equalization function didn't support an input of 16 bits. Then I performed a histogram equalization so that the corner points are detectable in the camera calibration OpenCV functions.

[mrcfschr]

Thank you I'll give it a try! this is what I've got so far https://drive.google.com/file/d/1rgc01WuRAIh_TM_ut0QdsQyjZ8_14m9z/view?usp=sharing

Parameter	PSI	Checkerboard	C++ RM Mapping
reprojection error	1.374427541024572	0.04400390079043265	?
Intrinsic parameters (camera matrix)	[[245.46107114, 0, 257.45927135], [0, 245.53169049, 257.08910023], [0, 0, 1]]	[[245.31616967, 0, 255.09594219], [0, 246.12117205, 247.46196344], [0, 0, 1]]	?
Distortion coefficients	[[-0.22265327, 0.03156156, 0, 0, 0]]	[[-0.300205711, 0.0969518716, 0.00391632399, -0.0000820810782, -0.0139891582]]	?
newcameramtx	[[148.94296265, 0, 256.84486832], [0, 148.97277832, 256.50581821], [0, 0, 1]]	[[141.71080017, 0, 251.40767011], [0, 143.26325989, 250.54294455], [0, 0, 1]]	?

*rounded a bit in the table for visual purposes but not in the actual calculations

Distorted	PSI	Checkerboard

The tracked tool does look slightly more aligned but still not perfect..

[mrcfschr]

The tracked tool does look slightly more aligned but still not perfect.. closing this for now.