marek-simonik / record3d

Accompanying library for the Record3D iOS app (https://record3d.app/). Allows you to receive RGBD stream from iOS devices with TrueDepth camera(s).
https://record3d.app/
GNU Lesser General Public License v2.1
379 stars 55 forks source link

Occasional lag in USB streaming #72

Open haritheja-e opened 11 months ago

haritheja-e commented 11 months ago

Hi,

We’re using Record3d for our project to video stream footage from a robot via USB to a NUC. When streaming video (at 30 fps) from the Record3d app using an iPhone 13 Pro, we notice at sporadic times a variable lag in the video stream (1-3 seconds). Are there any specific settings that can be selected to possibly reduce this lag? For more context, we notice this occurs more often when iPhone battery is low (< 20%), and have tried so on multiple iPhone 13 Pros. Could there be other factors possibly causing this lag?

Thanks!

marek-simonik commented 11 months ago

Hi,

I apologize for the delay in my reply. The behaviour you described sounds like a bug, so I cannot think of any way of mitigating it (i.e. ideally, the lag should not be there).

Based on your description, it sounds like the bug is more likely to be related to the iOS app rather than this library, but it might be possible that there is a race condition in this (record3d) library; this library receives the data stream in a separate thread. As shown in the C++ demo, the user of this library can process the incoming data either on the data-receiving thread (which should keep work to minimum to avoid delays) or on the main thread. Where does your code process the incoming data? Try comparing if the issues with lag go away when you process the received data directly in the stream.OnNewFrame callback.

I'd like to ask a few questions, if you don't mind:

  1. Do I assume correctly that the iPhones are not in Low Power Mode?
  2. Does the lag "go away" after a while? This is how I imagine the problem manifests: after streaming for some amount of time without a lag, you suddenly observe 1-second lag, but the stream gradually "catches up" and continues to stream without a lag.
  3. Does the lag occur in both the C++ and the Python demo or only in one of them?