AHAT Data

[kianwei96]

Hi @cgsaxner ! Thanks for sharing this repo, it runs nicely on my setup :) had no idea how to get a DLL up and running, so you're a life saver! If I understand it correctly, rig2world transform is stored in the ahat_receiver.latest_header right? Apart from that, I'm looking to get some additional information:

1. Active brightness buffer (512x512) for each recorded frame
2. AHaT extrinsics, which maps depth camera coordinates to rig (static transform, one matrix per streaming session)
3. AHaT look-up table, to map pixel to 3D vector (static map, one matrix per streaming session)

Actually I can grab 2 and 3 from previous recordings done with the StreamRecorder sample app, so if it's a hassle to grab them again, I can always bake it into the code!

How would I go about getting the AB image together with the depth image? Would I have to modify the DLL drastically?

Thanks :)

[cgsaxner]

Hi! Glad to hear that you find the repo useful. Yes, AHat cam2rig and look-up-table are obtainable from the stream recorder sample. this could, of course, be added to the DLL, however, since as you pointed out, they are static, I don't stream them to save the bandwidth.

Grabbing the active brightness buffer in the DLL should not be too difficult. You would have to modify the Streamer::Send() and add: const UINT16* pAbImage = nullptr; hr = pDepthFrame->GetAbDepthBuffer(&pAbImage, &outBufferCount); after the depht frame is grabbed via QueryInterface(IID_PPV_ARGS(&pDepthFrame));. Since the AB buffer seems to be UINT 16, same as the depth buffer, you should be able to use the same code for extracting byte data from the buffer and send it via WriteBytes().

[kianwei96]

Hi @cgsaxner ,

Yup, I got it to transmit both Ab and Depth in the same message, thanks for the tips! I noticed that there's a significant delay sometimes (up to a few seconds), and noted down the timings for a short streaming session here:

Am I right to say that this is because you opted for TCP instead of UDP, and due to a weak/intermittent network the packets "to send" from Hololens gets clogged up? I don't see how else the difference in timestamps (whether treating hololen's timestamps as ns or hundreds of ns) can be explained!

Look forward to your take on this matter :)

[cgsaxner]

Hi, good to hear you managed to get AB and Depth to stream!

Yes, trying UDP sockets instead of TCP is something I've had on my roadmap for quite some time, but have not gotten around to investigate any further. I also have the suspicion that something gets clogged up in the sending of frames, as on the Python side, nothing seems to block the receiving.

Another thing to try would be to somehow reduce the amount of data sent, e.g. by performing some sort of compression on frames before sending the byte data. For example, for the RGB video stream I discarded the Alpha channel to reduce the amount of sent bytes. Of course, this all largely depends on what you want to do with the data later on.

I also found that the frame rates of the research mode sensors are not constant, so sometimes the HoloLens just does not deliver frames for up to around 500 ms. Not sure what you need for your application, but this all makes applications which require some synchronization between different sensors very difficult.

[Lancename]

Hi @cgsaxner , Thank you very much for your project. I have successfully obtained the RGB and Depth data, but as you discussed, there is a significant delay in the data. I found a rather peculiar thing, when only receiving depth data, the delay is not very obvious, but once the hand appears in the field of view, it will cause a very obvious delay. I guess this is the delay caused by HL in recognizing gestures.

[cgsaxner]

Hi @Lancename, you're welcome!

About the hands in the field of view, I have actually noticed the same thing! Good to hear that I'm not the only one noticing this. As a matter of fact, I've noticed a delay / lag for all Research Mode streams as soon as the hands appear in the field of view. I'm also pretty sure this has to do with the hand tracking / gesture recognition, and I have not yet found a workaround there. I even tried to "fool" the tracking by wearing gloves, but even though this leads the gesture recognition to fail, the delay still happens. I assume that the tracking not only targets the hands, but the entire arms of the user. I'm afraid for now we just have to deal with that :/

[Lancename]

@cgsaxner
Hi, I'm so excited. I solved the problem of RGB image lag to a certain extent. The reason for the lag is that the sampling rate of the underlying image stream is higher than the transmission rate of TCP (I remember that the underlying frequency of RGB is about 30Hz, and the TCP test on the PC side About 15~20Hz). This means that with the longer the TCP connection, the more data in the HL bottom buffer area, the more serious the lag of the data received by the host computer, and the lag time is about 0.5 times the connection time. Therefore, there are two solutions: 1. Change to UDP transmission mode, but the transmission rate of UDP is best to be around 30Hz, or faster. 2. Slow down the data sampling and transmission of the bottom layer. Because I am not familiar with this kind of project, I tried solution 1 but there are always bugs. So I tried option two, and guess what, it worked! To be more specific, just add "Sleep(80)" to about 343 lines in VideoCameraStreamer.cpp, where 80 is about the transmission time of my TCP (WiFi6).

The same has also greatly improved Depth, but the effect has not yet reached my requirements. If you want to do the same in depth, just add similar code to line 187 of streamer.cpp.

[sysu17364029]

Hi @Lancename , I found that changing the value of 'minDelta' could help as well. In 'HL2RmStreamUnityPlugin.cpp', I set the value in : co_await m_videoFrameProcessor->InitializeAsync(2000000, m_worldOrigin, L"23940"); which makes Hololens send 5 frames per second. （I hope I understand your question correctly.）

[Lancename]

@sysu17364029 Hi, your understanding is correct, and you have provided me with a more effective code. Thank you very much. Many hands provide great strength！！！

[cgsaxner]

Thanks @Lancename for investigating and the update on this! As @sysu17364029 pointed out, the minDelta was added for exactly this reason, to achieve a better synchronization between RGB and Depth in the receiver. Obviously, by enforcing a maximal frame rate of 5 fps, you loose quite some information. Whether this is a big issue or not depends on the application, of course!

[dogadogan]

When we run deployed the project on the HoloLens 2, there is a huge lag. When we tried the approach by @Lancename, we were able to reduce the lag to 3-4 seconds. However, for our applications, we would need it to be <1 second (almost real-time). Is there a way to achieve faster communication? :) Could this repo be used over USB rather than WiFi, for example?

@sysu17364029, we also tried your approach but were confused where to add co_await m_videoFrameProcessor->InitializeAsync(2000000, m_worldOrigin, L"23940"); -- should we replace co_await m_pVideoFrameProcessor->InitializeAsync(m_pVideoFrameStreamer); (do you mean m_pVideoFrameProcessor instead of m_videoFrameProcessor)? When we used m_pVideoFrameProcessor, we got a syntax error for the use of InitializeAsync (having three arguments). We would appreciate it if you could clarify!