Colored point cloud offset / mismatch

[cgsaxner]

Hi! I made some recordings of PV and DepthAHAT streams with the StreamRecorder and processed them with process_all.py to obtain colored point clouds. In those point clouds, there is a quite large mismatch (I would guess around 2 cm) between depth and rgb information, as can be seen from this example:

With the HoloLens1 using my own code, the matching is much better:

Is this behavior expected, or is something wrong with my recordings? Is there anything I can do (except for adding a fixed offset) to obtain a better matching between DepthAHAT and PV?

Thanks!

[fbogo]

Currently, the matching between depth and PV frames based on timestamps is quite naive (given a depth frame, the closest PV frame in time is picked to get RGB values). So any motions could cause misalignments. In order to try identify if this is strictly a problem with AHAT instead, could you please confirm if you observe the same behavior when obtaining point clouds from Long Throw Depth?

[cgsaxner]

I see, thanks for your reply. Here is a point cloud from Long Throw Depth:

The problem looks quite the same, the RGB information seems to be shifted to the left wrt. the depth information. After looking at some more frames and recordings, it looks to me like a systematic error, since the shift is always approximately the same, regardless of motion direction etc. So maybe some of the camera parameters are not quite right?

[kysucix]

Hi cgsaxner,

I didn't experience such a systematic bias, I even tried with a fire extinguisher as well:

Which windows version are you using?

[cgsaxner]

Hi, this looks indeed much better than my results. I'm using Windows 10 Pro Version 10.0.19041 on my PC, the HoloLens 2 runs build 19041.1384. Thanks!

[kysucix]

Could you try to make sure you're running the latest available build please?

[cgsaxner]

Sorry for the late reply, I did not have access to the device for the past days. After upgrading to the newest build on the dev channel and pulling the latest version from this repository, the problem still persists. Long throw depth: AHAT depth: I also noticed that there seems to be a new problem with the AHAT reconstruction now, where some points seem to be wrongly reconstructed near the camera (red circle). These points are present in every AHAT point cloud I looked at, in some cases much more severe than in this example.

[wanglolly]

Sorry for the late reply, I did not have access to the device for the past days. After upgrading to the newest build on the dev channel and pulling the latest version from this repository, the problem still persists. Long throw depth: AHAT depth: I also noticed that there seems to be a new problem with the AHAT reconstruction now, where some points seem to be wrongly reconstructed near the camera (red circle). These points are present in every AHAT point cloud I looked at, in some cases much more severe than in this example.

Hello! I have exactly the same problem in the AHAT depth reconstruction currently. In each frame, there are some wrong reconstructed point near the camera itself, this is somehow weird and will be a seriously problem when preserving all the frames. Does anyone have any idea how to solve it? Thank you very much :)

[estherlxd]

Hi, I am also experiencing the same issue. I fixed my Hololens 2 on a tripod to avoid the camera motion, and used StreamRecorder App to capture PV and Long throw frames, so the RGB data should exactly match the 3D point clouds. But different from what I expected, after running process_all.py to obtain colored point clouds, I observed that the RGB information seems to be shifted to the left wrt. the depth information. And with depth distance between HL2 and objects increased, the size of the offset also becomes larger. With distance 40cm: With distance 90cm:

Does anyone have any idea how to solve it? Thanks!

[Fouksi]

Hi,

Did anybody managed to solve this issue? I've experiencing the same thing.

Thanks

[cgsaxner]

Hi, I eventually managed to kind of overcome this problem by calibrating the HoloLens cameras myself and by replacing the camera parameters loaded from files in the reconstruction code with my own. It seems like the on-board calibration of the HoloLens is just not very accurate. The calibration is quite challenging though, and it still does not look perfect. I used this toolbox: https://github.com/RobVisLab/camera_calibration

[Fouksi]

I kind of fixed it manually by editing the rotation vector, adding a code line to the function 'project_on_pv' at the 'utils.py' script: [...] rvec = np.zeros(3) rvec[1] = 0.025 tvec = np.zeros(3) [...]

[kysucix]

It looks like this is fixed in #104 please try with latest master

[chenritiana]

hello,how did you get the points with color?I'm interested in this