microsoft / HoloLensForCV

Sample code and documentation for using the Microsoft HoloLens for Computer Vision research
MIT License
472 stars 156 forks source link

Mapping depth to VLC cameras #138

Closed mgeorgec999 closed 3 years ago

mgeorgec999 commented 3 years ago

Hello everyone.. I am relatively new to this platform. Will add projects to the repository soon.

I have been trying to map the depth frame to the VLC frames for sometime now and realized i have a few gaps in my understanding.. There are a few projects related to hololensForCV which maps depth frame to PV which i have been able to reproduce. Thank you! It seems that the only missing matrix to reproduce this for VLC frame would be the VLC camera projection transform.

Any suggestion by anyone on how to recover this transformation matrix?

I tried the following: a. Tried to recover the matrix by finding correspondence between the pixel coordinates and the world points (.bin file from the recorder tool). I tried using the estimateCameraMatrix function in matlab for this.. Is the Worldpoint just the inverse projection transform of the pixel coordinates? Tried to reproduce this for the PV frame and there was absolutely no match between the estimated projection matrix and the actual projection matrix in the pv.csv file. b. Tried looking up the recorder.py sample which lists the f, cx and cy for the VLC frames.. but did not figure out how to go from there. Does anyone have experience with COLMAP? Is there a way to recover projection matrix from it?

Any pointers, suggestions would be much appreciated! Thanks..