search

marek-simonik / record3d-simple-wifi-streaming-demo

MIT License
15 stars 6 forks source link

WiFI Streaming works but shows some RGB not LIDAR next to RGB #4

Closed sleeplessTLV closed 3 weeks ago

sleeplessTLV commented 3 weeks ago

Hi, I can stream just fine to my pc , but what I see on both PC and the app is NOT like the image in the github explanation. Instead of two images side by side, one LIDAR (color coded) and one RGB, I see just one RGB image, with some 'hints' of LIDAR readings as it does not look like regular RGB (see attached) I use latest Chrome on a Windows 11 machine, iPhone Pro 14, both on same WiFi network of course. lidar3d_iphone

marek-simonik commented 3 weeks ago

Hi,

firstly, thanks for using Record3D.

I could not replicate the problem you are describing when running the simple RGBD demo (this repo).

The screenshot you attached looks like it was taken while running the other, more complicated Three.js 3D demo, which creates a colored point cloud out of the RGB and color-encoded depth map video stream.

The easiest way to see the raw side-by-side RGBD stream is to:

  1. Enable Wi-Fi streaming in Record3D in the Settings tab.
  2. Go to the Record tab and press the red toggle button. The app will show the IP address of your iPhone/iPad.
  3. On your computer: open your browser and enter the IP address of your iPhone which you saw in the Record3D app. The Record3D app runs a webserver on port 80 of your iPhone/iPad, which serves the simple streaming demo (contents of this repo) for convenience.
  4. You should see a simple website with the title "Record3D live-streamed video from your iPhone/iPad" containing an input text field pre-filled with the IP address of your iPhone/iPad. There should also be a "Press to start streaming" button. Please press the button to start streaming — you should see the raw RGBD stream with side-by-side Depth and RGB images.

Please let me know if this helped or if there are still problems with the stream. Thanks!

sleeplessTLV commented 3 weeks ago

Thank you for the quick response! I first tried the 'simple app' as per these instructions, and the iPhone did report it is connected and streaming the RGBD video, but the screen remained always blank (see sample screenshot). That's why I switched to the Three.js app (https://192.168.xx.yy/3d-demo/) which does work, but does not output the video as I wish (the simple version). Any ideas? lidar3d_iphphone2

sleeplessTLV commented 3 weeks ago

OK an update - with Microsoft Edge the simple demo DOES work! So it is a Chrome browser issue as you mention in the github documentation. Is there a way to bypass it? It is not critical since I can use Edge, just wondering. Follow up quick one - is there a way to make the LIDAR display use grayscale instead of colors? or control it in some other way? Thanks!

marek-simonik commented 3 weeks ago

That's an interesting observation — perhaps Chrome does now use more security restrictions compared to Edge? Unfortunately, I am unable to install Chrome at this moment to test this myself, but may I ask if starting Chrome with the same parameters as mentioned in the section Option 3 of the following comment makes the RGBD stream visible: https://github.com/marek-simonik/record3d-simple-wifi-streaming-demo/issues/2#issuecomment-1287568104 ?

Regarding your second question about grayscale depth images: unfortunately, the side-by-side RGBD stream needs to be encoded using the HSV-based color encoding, so there is no option to make it grayscale. However, you could post-process each frame of the received video (the stream is available in a <video> tag, as can be seen from both demos) and convert it into grayscale in browser.

sleeplessTLV commented 3 weeks ago

Thank you, that is indeed no problem. But my intent is that the grayscale will correspond to distance/depth reading - e.g. bright values to close objects and darker as they become more distant, on a linear scale. Is that what conversion to grayscale will accomplish? thanks

marek-simonik commented 3 weeks ago

If you only want to obtain the (approximate) depth in meters, then you don't need to convert the color-encoded depth into grayscale (in fact, converting the RGB color directly into a grayscale value by averaging the 3 color components would not give you a depth value). Please see the description of the color encoding in the README of this demo (the section RGBD Video Format) to understand how depth can be decoded from color.

And here you can see an example implementation, which converts a RGB color-encoded depth pixel value into absolute distance in meters.

If your workflow requires depth maps encoded into grayscale images, then you will first need to convert the color(RGB)-encoded depth values into absolute depth in meters (using the guide linked above), and then convert the depth values in meters into grayscale pixel value in the range [0; 255] by normalizing them.

But note that conversion into grayscale depthmap will discretize the continuous range of depth values [0.0; 3.0] meters into just 255 discrete uint8 values. (This is why Record3D encodes depth into colors — so that there is much more discrete values, making the decoded absolute depth more accurate).

sleeplessTLV commented 3 weeks ago

Thank you, very clear and I verified it worked for me.