Hololens 2 VideoEncodedImageCameraView goes down to 1/s on slow networks

[cwule]

To recreate issue: Run HololensCapture sample and HololensCaptureServer on fast local network (e.g. Netgear NightHawk router) and the VideoEncodedImageCameraView streams at about 30/s.

Run HololensCapture sample and HololensCaptureServer on slow local network (e.g. via Google Pixel hotspot) and the VideoEncodedImageCameraView streams at about 1/s.

The other streams (depth,...) stay about the same with 5/s.

[AshleyF]

You could try reducing the amount of data being transmitted by disabling streams in which you're not interested and/or reducing the resolution or framerate. By default, it may be attempting to transfer a significant amount of data which could very well be an issue over a slow network. For some of our work, in fact, we use a direct USB connection between the HoloLens and the capture server.

To get an idea of the amount of data being transmitted, in PsiStudio you can right-click on a stream in your collected data and choose Show Stream Info > Throughput (bytes per second). The video and depth streams constitute the bulk of the data, but the left/right grey cameras (which are enabled by default) are ~200K/s each, audio is ~170K/s, IMU streams combined are ~150K/s, etc. They may be disabled with the various IncludeFoo flags in the capture app.

You could also reduce the resolution (PhotoVideoImageWidth/Height) or frame rates (be sure to check here for valid combinations).

A more advanced option would be to experiment with DeliveryPolicy settings on the video streams. It is possible to control dropping/queuing of messages to ensure that frames don't drop even when they cannot immediately be delivered over a slow network connection. However, this option requires very careful consideration of the various tradeoffs and may cause unexpected behavior such as exhaustion of memory on the device as queues fill and long shutdown times while queues drain, etc.

Hopefully this helps. Let us know how it goes for you.

[cwule]

Thanks a lot, the PsiStudio tool profiling tip is very useful showing ~20M/s for the RGB vs 900K/s for the depth camera with even less for the other streams. Didn't realize that the difference is that big. I suppose decreasing the jpeg compression quality should also help in that case. Yes, the DeliveryPolicy queuing seems to be tricky if data comes through at such a high rate.

Thanks a lot btw, the repository is so valuable for a variety of applications!

[AshleyF]

Ah yes, JPEG encoding is another approach. As you say, you can control the GrayImageJpegQuality and InfraredImageJpegQuality. However, you'll notice that there is no such setting for video/preview. These already come as NV12-encoded (12-bits per pixel) directly from the hardware. You could experiment with decoding and re-encoding as JPEG to save bandwidth (at the cost of CPU-time). For example:

var jpegEncodedVideo = camera.VideoEncodedImageCameraView
    .Decode(new ImageFromNV12StreamDecoder(), DeliveryPolicy.SynchronousOrThrottle)
    .Encode(new ImageToJpegStreamEncoder(0.8), DeliveryPolicy.SynchronousOrThrottle);