Syncing 3D depth map information with Unity world

[jdagdelen]

I wanted to open up discussion on where we want to go with this project. I think there are two directions:

2D scenario:

In this case, the views from the cameras are streamed from the car and we insert objects into the video using the AR approaches we've covered in class. I don't really see a reason to put a driver in VR if this is the approach other than "immersion", because we'd be recreating a 2D stream on a virtual 2D screen, which is pretty much what Nintendo are doing in the switch AR Mario Kart.

3D scenario:

Figure out how to stream the depth map data from the RealSense camera or use depth mapping algorithm on a 2D video stream and place the 3D geometry from the real world into the player's environment. This has been demonstrated recently, and you can see some example results here.

Encoder:

https://github.com/bmegli/realsense-network-hardware-video-encoder

Decoder:

https://github.com/bmegli/unity-network-hardware-video-decoder

This adds ~100ms latency, which is obviously too much for realtime streaming in VR. However I think there could be a middle-ground where we use the AR maker localization data to infer the "real time" positions of the points in the point cloud and update the geometry in Unity occasionally. Unless the car is in a spin/skid, its instantaneous velocity will always be approximately parallel to the normal of the depth map, so I think we could get away with this here.

I personally think this is the more challenging + interesting option of the two, and it would be pretty sweet to be driving a car through a depth-mapped image, sort of like a sonar signal or something.

[jdagdelen]

Another resource for using GStreamer with the realsense to stream depth map data, implements Intel's whitepaper on the same:

https://github.com/WKDSMRT/realsense-gstreamer

https://github.com/johnny-wang16/rs-udpstreaming

[jdagdelen]

Quick update: I was able to get depth map streaming working over ethernet LAN using this implementation: https://github.com/jdagdelen/rs-udpstreaming. I haven't tried wifi yet, but I think we're off to a good start.

[jdagdelen]

Actually, it looks like Intel actually enabled this as part of librealsense2 so we can probably just go with their implementation of the depth-to-color. Reference implementation for recovering depth from color here: https://github.com/TetsuriSonoda/rs-colorize/blob/master/rs-colorize/rs-colorize.cpp.