Beta 13 Release Reintroduces NHL Commercial Issue

[SoSJames]

Hello again and happy new year!

I installed the Beta 13 release and used it to watch the NHL Stars vs. Ducks game last night. It looks like the switch to the custom HLS proxy brought back the skipping/commercial issues from issue 29 way back when. Except now the behavior's a bit different. Now, the commercials go crazy like the proxy can't figure out which ones to play. You'll see about 5 seconds of one, then it switches to something else, then back to the first, and so on. Eventually, the game comes back, but starts freezing/skipping like it did when Streamlink was in the mix. I can't easily see the contents of the m3u8 when it happens, so I don't exactly know how they're injecting the commercials. If it's an X-Discontinuity like I previously suspected, perhaps it's possible to use the proxy to filter out the commercial segments and replace them with ESPN+'s "Commercial break in progress" video. If you can capture that and store it locally to substitute, perhaps it won't break the stream (since it's the same resolution and framerate as everything else). Anyway, I've reverted to the previous version in the mean time. Hopefully there's some way to fix the issue, since your proxy seems far more efficient.

Thanks!

[m0ngr31]

I'm guessing this is on ESPN+ right? I live in a region that doesn't get ad breaks on ESPN+ so it might be tough to investigate. I'll look into it though

[SoSJames]

Yeah, it's ESPN+, and I've only noticed it on a NHL games. A VPN into the US east coast might make troubleshooting easier. Proton's free tier should do the trick if you don't have one. In the mean time, I'll try and see if I can capture an m3u8 from one of the commercial breaks to see what's changing. Perhaps if I can identify what's happening I can give you some more info to go on. In any case, thanks for checking the issue!

[dozer1dave]

I see the same thing described regarding commercials with version 12 when watching ESPN+ . I am currently seeing while watching a NHL recap show called "The Point". Happy to grab any additional info if you tell me what to grab.

Video of commercial here: https://drive.google.com/file/d/1fGkqmntKr_VATUgCGPG0sluCQ8YKhVaj/view?usp=sharing

[m0ngr31]

Are you watching through Channels or what? I'm curious if it's having an issue with the discontinuity tags where before ffmpeg wasn't passing them through before... :thinking:

[dozer1dave]

Yes, watching thru channels.

[SoSJames]

To chime in, I see the same behavior when watching the m3u8 on Jellyfin, VLC, and MPC-HC. The FFMPEG build I pointed you towards last time this came up had a patch that could handle discontinuity tags by re-clocking the segments so they didn't disrupt whatever player you're using. It prevented non-compatible players from choking on the ad segments. That is, of course, assuming I'm right about how they're injecting the ads. This is a common problem on plenty of HLS projects. From the discussion on Streamlink's Github:

EXT-X-DISCONTINUITY indicates a complete reset of the stream, with a new stream following afterwards.

This is why it makes many players fail, because they see the tag as a failed segment and try to re-read it, hence the skipping. Most players never recover from the tag. Some just quit entirely. That's why I thought it might be possible to filter those segments using the proxy and replace them with something more stable, like the repeating ESPN+ commercial break in progress screen.

[SoSJames]

Also, if it helps, I believe the Streamlink Twitch plugin implements something in its parser to remove injected ads and deal with the discontinuity tag. You can find the code here: https://github.com/streamlink/streamlink/blob/master/src/streamlink/plugins/twitch.py Maybe it could be modified to work similarly in your proxy.

[m0ngr31]

I'm guessing you have similar issues on things like Pluto.tv then?

[SoSJames]

Yes, exactly. Pluto.tv does the same kind of thing. For that, there's a program called Cabernet that includes a proxy/filter to deal with the ad problems. You can find it here:

https://github.com/cabernetwork/cabernet

[m0ngr31]

Hmm, I'm haven't seen that before. I wonder if that would fix this as well?

[SoSJames]

I suppose it's possible, but since I haven't yet seen the contents of a commercial-including m3u8, I'm basing all of this on educated guesses.

[m0ngr31]

If you're wanting to see the m3u8 coming through, you can use something like this: https://chrome.google.com/webstore/detail/native-hls-playback/emnphkkblegpebimobpbekeedfgemhof

You can open the dev tools and see the network requests come through in real time. This is part of what I use to debug things.

[SoSJames]

Yeah, I mean I've got to capture one of the m3u8s as text during a segment that contains a commercial. If I'm not mistaken the problem only occurs during live broadcasts, so I can't use a replay. Unfortunately I'm not always free for large chunks of time when a live NHL game's on. If I can grab one of those, I could at least see the structure of how they're injecting the ads, i.e. discontinuity tags etc. That might make constructing a filter easier. I also might be able to find one of their generic commercial break slates as raw .ts files that you perhaps can use to substitute for the broken segments. Sadly, I'm not a programmer, just a network engineer, so I understand the concepts but can't provide much direct help.

[m0ngr31]

I totally understand. I've been working at streaming companies for years now, but I'm certainly no expert with HLS so I'm happy to get help from everyone who can.

I'll experiment with https://www.npmjs.com/package/media-stream-validator and see if it can give me some hints

[SoSJames]

Another thought that I had is that you might be able to parse the m3u8 in the proxy, look for the commercial break, and tell the proxy to reload its process going into and out of the breaks. If it happens fast enough, the player end may not notice anything is happening and continue to play. Might require a buffer of some kind, though. In any case, thanks for all the effort. I seriously appreciate this program since it makes it simple to watch sports within my Jellyfin instance. I'm happy to stick with the previous version if/until there's a solution.

[m0ngr31]

I played with it a bit this afternoon and I can see it happening on a few events. Pretty obnoxious but it didn't mess up the stream after the break. That could be player dependent though. I'll experiment with other players and with Channels soon to see how bad it's breaking those

[dozer1dave]

For me, on channels it screwed up the commercials but that was it. The programming I cared about was fine. For me, this is a mild annoyance.

[blackthroat]

Watching the Sharks vs Ducks game right now through Channels (Browser and Apple TV) and experiencing the same issue with commercials hopping around, as well as playing the first few seconds of the same commercial 5+ times in a row. "When you buy 10 tickets on Vivid Seats, you get the 11th ticket free." Agreed that it is pretty obnoxious and that it will absolutely drive me insane over time. I'm more than happy to provide any additional information or test any fixes down the line. Thanks @m0ngr31!

[m0ngr31]

I figured it out 😀

I have 2 other fixes I need to make and I'll release a new version

[m0ngr31]

I pushed a new container to Docker just now. Let me know how that works for everyone!

[SoSJames]

I'll update my container and report back. Thanks again for the quick response, as always, you're the best!

[dozer1dave]

Just loaded v14 and for me the commercial skipping seems fixed now. What I am seeing is when I launch an event the "starting soon" screen followed by either a "Playback Failed (0)" or "Timeout after 32s" error. I press select on my remote and it starts the stream. I have seen these errors previously but they were somewhat sporadic. I am using Channels on a FireTV Cube. See log.

eplustvv14_logs.txt

[dozer1dave]

Without a VPN my logs are very clean but I still get the "Playback Failed (0)" when starting each event. eplustv14novpn_logs.txt

[SoSJames]

Similarly, I get the stalled playback. I'm wondering if your slate (event will start soon) is the same resolution/framerate as the broadcast. If not, that could cause such an issue. However, the commercials seem like they're working and the player's no longer dying going into and out of the breaks. I'm content to relaunch the channel to get around the startup stalling.

On a side note, I looked at your fix and it looks like you're rearranging the segments to get around the problem, no? If that works to fix the discontinuity problem you may have created a proxy that other projects could apply on hundreds of other streams. Awesome!

In any case, I believe the original issue is fixed here, so thanks for all the hard work. I'll leave the issue for you to close since I'm not the only one chiming in here.

[m0ngr31]

I'm not actually rearranging the segments, I'm just not proxying the commercial breaks. Interestingly enough the commercial breaks don't use the discontinuity tags. I'm going to make a new issue to track the issue with events having to launch twice to get working. So frustrating! I thought I had it this time, but its totally dependent on the player so I'm trying to figure that out.