Trying to figure out poor quality playback

[AlexSapple]

Hi!

great library - ability to play H264 raw is fantastic!

just a question (and i'm quite new to working with video) and this may be a bad question as it's a bit vague - I'm trying to playback a live stream in browser - it works, but the quality is not good. Whilst this is likely to have nothing to do with JMuxer - wonder if I can use this to get some advice on troubleshooting.

I'm getting frames of H264 back (initially over http upstream) and then pushing those frames on to a websocket that jmuxer is attached to in a browser window (yes, there is a possibility that latency is an issue with multiple network hops in play).

I get periods of good playback and periods of severely artifacted playback. Things like colors breaking down (things turning green), pixelated, sometimes a kind of wire frame like display and sometimes I might get things like top half of image looking good and bottom half looking pixelated.

things I've considered (but not sure how to confirm or how to resolve).

partial frame responses (particularly on keyframes since they hold more bytes) - since my starting point is an upstream http stream - each time i "read" from that upstream stream, I may not have a complete frame (http chunked encoding) - there is a (likely) chance I may push an incomplete frame down the websocket. I have tried playing around with this - but it's remarkably difficult to handle :) - when I see half and image looking good and the other half bad, it makes me think of this!

fps - I'm not setting a frame rate because It's dynamic in the feed - and I don't do anything to feed frames at a set pace (should I?) - at the moment, I just feed frames as soon as I get them.

key frame interval - this is dynamic in my case, but I can request upstream to send key frames on command.

[AlexSapple]

update:

I wrote some code to ensure that I wouldn't send partial frames down the websocket. This did change things. I believe the playback looks much better now, but I'm still having issues.

• now I'm getting really big delays before playback start - as if it hasn't hit a keyframe (I need to look at that from the stream end).

• after maybe 10 seconds of clear playback without any artifacts - it then scrambles and doesn't recover. Not sure why this is, but I'm guessing after doing some more research that this could be related to incorrect duration (I think my feed has a dynamic frame rate and therefore It makes sense that I need to pass this duration value). I will update once I've handled this case.

[samirkumardas]

1. Glad to hear you are unveiling the tangle gradually :)
2. Along with keyframe, it requires SPS, PPS to start playing. Therefore, make sure your stream contains SPS, PPS frequently so it can start at anytime
3. The main reason behind the artifact is missing/partial frames. Grab buffer of h264 for some time say 30 seconds, play it with https://samirkumardas.github.io/jmuxer/h264_player.html If it played well, it must be missing/partial frame issue.

By the way, I will push a commit to mitigate partial frame issues for fixed fps soon.

[AlexSapple]

Update 2:

I have now managed to feed in duration. This was more difficult from my end because my stream was just the raw data and didn't immediately expose duration or frame rate data. It was a slightly painful process but I'm quite happy with the final result!

• I'm only feeding duration data when it changes - it seemed to be really slow if I fed the duration on every call to feed - is that valid? i.e. does JMuxer remember the rate from the previous call to feed? - playback looks good now, so I'm hoping that it does.

• I'm still having issues that after some seconds the playback just stops (and sometimes scrambles) - the debug console clearly shows data is still being fed - but playback freezes.

• I'm still having issues with start of playback. I may have a race condition upstream that is cleaning up frames - whilst I haven't proved this, it could potentially be cases where frames might be lost as a result (in theory this shouldn't happen... but bugs happen :) ) it might explain why a valid key frame frame to begin playback isn't [always] hit). This same cleanup may mean occasionally losing a frame. It's going to be tricky to prove that theory :)

• if I try to feed individual frames down the socket (so literally 1 at a time rather than a chunk of complete frames) - I seem to be hitting the following sequences of errors: "Error occured while appending video buffer - InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source."

followed by "mediasource is not available to end"

I don't seem to see these errors if I slow the feed down and say aim for a chunk of 10 frames (I would aim for complete GoPs - but I don't think I have a reliable way to identify keyframes in a sequence in order to make a valid feed of GoPs). Also since i'm working with a live stream - I can't slow this down too much or risk getting behind live.

• just a question - and i'm really sorry if this is a stupid question. The duration is an integer value of milliseconds but a typical frame rate or 15 or 30 doesn't split to a whole millisecond duration. The difference is clearly a tiny fraction but am I supposed to manage this in some way? (at the moment I just feed duration 33 or 66 etc).

• I tried taking a file dump of the frames and play in the link you sent. Unfortunately playback did not start :( but the same dump did play in VLC (I'm not making a comparison - I'm just saying that to indicate that the data was not corrupted - VLC is certainly weird in so much that it seems to be able to play things by magic! again, maybe it was missing a key frame to start).

Still some way to go on this - but certainly the artifacts appear to be much less than before.

[samirkumardas]

Replied after having a glance. Will check it later again, just curious about the h264 file that played in VC player but not in jmuxer h264 player.

Can you share the h264 file that is playable in VLC but not in jmuxer?

If you okay with a fixed fps or don't have a variable frame rate, you don't need to provide duration while feeding. You can simply set an fps during instantiating jMuxer.

[AlexSapple]

thanks for the reply!

sure - I've attached the file (it's essentially a webcam feed that's re streamed) - and if it plays you will see my face with a silly smile! - sorry about that :)

SampleRAWVideo.zip

[samirkumardas]

I was abled to see your smile face! Firefox can play without issues. Not sure why the chrome that I always get a favourable browser failed here :) Interesting is that it plays if you clicked play button several times haha. Seems like it was waiting for more buffer

[AlexSapple]

Update 3:

I think I've now got reliable playback now. I managed to improve things again by tweaking how the stream is sent from the upstream server. It's not perfect, but it's much better than it was (and as a beginner - I've mostly run out of ideas :) I'm going to close this one now).

thanks @samirkumardas - great library and I appreciate your comments.