search

samirkumardas / jmuxer

jMuxer - a simple javascript mp4 muxer that works in both browser and node environment.
Other
555 stars 112 forks source link

Latency #7

Closed aaronovz1 closed 6 years ago

aaronovz1 commented 6 years ago

Hello,

Firstly, your project is the only thing I have managed to get working for the scenario I need. Basically I am encoding camera frames into H.264 using lib/ffmpeg in C/C++ and pushing those over websockets to a browser client. I originally tried to push mp4 container data directly into MSE but with no luck so far.

The problem I am facing is there seems to be too much latency (even if I am directly connected over Ethernet). I am not sure if the delay is on the C++/encoding side, websockets, or the remuxing/browser side.

Do you recall what kind of latency you were able to achieve with jmuxer?

PS. I have set flushingTime to 1ms which helped from the default of 1500, but there still seems to be at least 1 second of latency or more (I am yet to profile to get actual value, I am just going off the visual latency I see).

samirkumardas commented 6 years ago

Playing won't be started until it get an IDR frame (I -Frame). Look likes you have at least 1 second gap in between two IDR frames. You can try reducing that gap.

aaronovz1 commented 6 years ago

I did have key frame/IDR set to every 60 frames. Changing it to every 15 frames only gave the benefit of the live stream starting a little sooner. There is a still a noticeable delay when moving my hand in and out of the camera view. Is the delay caused by MediaSource requiring a certain amount of data before it starts playing?

samirkumardas commented 6 years ago

If it is not network latency, delay most probably caused by C++/encoding side.
In jmuxer end, only flashTime can affect on latency and I have not get any noticeable delay caused by MediaSource itself.

In our case, we had also delay approximate 2.5 seconds.

  1. More or less 2 seconds delay from our encoder side.
  2. WebSocket takes some time to establish (approx 500ms) from constructor calling to first byte.
aaronovz1 commented 6 years ago

I found the problem. There were two factors contributing to the latency:

  1. There was intensive code being run between getting a camera frame and pushing it over websocket
  2. The H.264 profile causes noticeable delay. We were using High by default but after switching to Baseline the latency was significantly reduced.