Streaming frames from rendering

[hrvthzs]

I am encoding raw h.264 frames using the NVidia Video Codec SDK on the server side and decoding them with jmuxer on the client side. Seems that, sent frames are not rendered immediately, they are always a 4 frames behind. This is very annoying when interacting with a 3D scene. Is there a there a way to encode and show the current frame immediately? Thanks :)

const jmuxer = new JMuxer({ node: "player", mode: "video", debug: true, flushingTime: 0 });

[sanalpencere]

Hi @hrvthzs,

Can you give us some details about that? Which browser are you using? I have a similar problem with Chrome on Windows. Also, If it is chrome can you disable hardware-accelerated decoding from chrome://flags. With that everything works fine on our side.

[jbrownsw]

I might be able to offer some insight here as I was just stepping through this troubleshooting a problem we're having. The example from our data is that we send a frame at a time of live video as it comes in. So the first frame is an I frame with SPS, PPS, and IDR frame. The feed function takes this and ends up putting the SPS and PPS in the slices variable leaving the IDR in left. The call to getVideoFrames obviously finds no frame in just the SPS, PPS data so the first call basically does nothing other than put the SPS, PPS in pendingUnits and the IDR in remainingData. The next time around we give it a non-IDR frame and this time the IDR is in slices so it gives it to getVideoFrames. Unfortunately getVideoFrames checks to see if it's a vcl is in pendingUnits THEN puts the IDR in units. Since there was no vcl in units prior to the IDR being added getVideoFrames again returns no chunks back. If the if statement was the last thing in that loop it would have pushed a frame sooner and only been 1 frame behind at this point. Instead we get to the 3rd frame our software passes through feed (another non-IDR). This time all the necessary information is in pendingUnits and it knows it has a vcl so when the 3rd frame gets passed into getVideoFrames it finally returns chunks and passes them to remux on the third frame.

I know the original commenter said 4 frames. Perhaps there's another scenario where this could be 4 or perhaps there's another 1 frame delay somewhere after the muxing. I can certainly reproduce it not decoding the 1st frame until the 3rd frame is passed in.

[hrvthzs]

hi @jbrownsw, @samirkumardas you are right it was 3 frames, it am using NvEncoder from NVidia Video Codec SDK, which had and extra output delay to ensure encode and graphics can work in parallel. But I still have the 3 frames delay, which is not happening when using Broadway-player (it has other issues though). Would be great to find a solution to this.

@sanalpencere I have tested different browsers, Edge and Firefox need 3 frames, but for some reason Chrome needs 8 with acc. decode, and 3 if disabled.

I am using NV_ENC_TUNING_INFO_ULTRA_LOW_LATENCY and NV_ENC_H264_PROFILE_BASELINE_GUID.

[sanalpencere]

@jbrownsw @hrvthzs

Thank you so much for your replies. If I found any clue about the solution. I will write it down here.

[meiguiyisenluo]

Has this problem been solved? I had the same problem in windows chrome v103.0.5060.66. did it lost the first pps and sps? And it will cause the parseNAL fucntion and H264Remuxer.readyToDecode is false until the next pps and sps are feeded.

[meiguiyisenluo]

I add some code in /src/jmuxer.js，it solved my problem for now, but I don't know if that's right

[nnnpa31]

Since you mentioned "4 frames" and Chrome may have a 4 frames buffer in non-live mode, the following may help to resolve this issue: https://stackoverflow.com/questions/36364943/frame-by-frame-decode-using-media-source-extension