Twitch.tv embedded ads (last update: 2021-11-21)

[bastimeyer]

Since we're seeing reports of new changes on Twitch regarding the embedded ads situation (once again), I feel the need of opening a new thread before it gets too messy again with the issues on the issue tracker here. I will update this post once we've got new informations, so that the current state can be found and read easily without digging through issue threads.

---

Edit 15 (November 2022)

See the new thread: https://github.com/streamlink/streamlink/issues/4949

Edit 14 (2021-11-21)

Updated the stream access token request parameter again to reduce the number of preroll and midroll ads. See #4194 Published in the 3.0.2 release (2021-11-25)

Edit 13 (2021-10-19)

Twitch has made new changes and embedded ads are back again and don't get filtered out correctly. See #4106

Edit 12 (2021-10-10)

New device ID headers implemented for preventing ads: #4086

Edit 11 (2021-06-21)

Re-pinning the thread, as it's still relevant.

If you don't want to see ads on Twitch, or if your player can't handle the stream's discontinuity between the ads and regular content and then crashes (eg. VLC), then set the --twitch-disable-ads parameter, which will filter out ads and pre-roll ads and will pause the stream output during that time.

CLI documentation: https://streamlink.github.io/cli.html#twitch

Streamlink Twitch GUI https://github.com/streamlink/streamlink-twitch-gui/wiki/Embedded-ads

Issue changelog

### Edit 10 (2020-12-23) Streamlink `2.0.0` has been released yesterday with the workaround from edit 9, but it doesn't seem to be working reliably anymore. Ads can still be filtered out via `--twitch-disable-ads`. ### Edit 9 (2020-11-28) Added a new access token request workaround in #3373 which again prevents ads from being embedded into the stream. (additional note: the access token requests have been moved to the GQL API already in #3344). Like usual when there's a new change, you can install Streamlink from the master branch, or wait for the next nightly installer to be built if you're on Windows, or wait for the next release. ### Edit 8 (2020-11-05) The ad prevention request parameter workaround has stopped working again, as predicted, and the ad "placeholder" screen seems to have been replaced with real ads now :boom: Filtering out ads via `--twitch-disable-ads` is however still working. ### Edit 7 (2020-10-28) A new ad prevention has been discovered (#3301) and has been merged into `master` on 2020-10-28. If you don't want to wait until the next release (`2.0.0` - release date unknown), you can [install Streamlink from the master branch](https://streamlink.github.io/install.html#pypi-package-and-source-code), or apply the small change yourself (cf978c5), or as a Windows user, [wait for the next nightly build of the installer (>= 2020-10-29)](https://github.com/streamlink/streamlink/actions?query=event%3Aschedule+is%3Asuccess+branch%3Amaster). If you're using the Twitch GUI, you will have to use a debug / pre-release due to other breaking Streamlink changes in regards to the upcoming `2.0.0` release: streamlink/streamlink-twitch-gui#757 ### Edit 6 (2020-10-18) [`1.7.0`](https://github.com/streamlink/streamlink/releases) has been released on 2020-10-18. ### Edit 5 (2020-10-03) #3220 has been merged as well, which should prevent the embedded ads again (for now (edit: broken again)). ### Edit 4 (2020-10-03) #3213 with the new ad filtering implementation has been merged into the master branch. You can install Streamlink from there, [as described in the docs](https://streamlink.github.io/install.html#pypi-package-and-source-code), or if you're on Windows without a Python environment, wait until the [2020-10-04 nightly installer](https://github.com/streamlink/streamlink/actions?query=event%3Aschedule+is%3Asuccess+branch%3Amaster) has been built. ### Edit 3 (2020-10-03) Removing the `platform=_` parameter again from the stream access token API request once again results in no ads being embedded into the stream. This is a game of cat and mouse at this point, which is ridiculous. https://github.com/streamlink/streamlink/issues/3210#issuecomment-703022351 ### Edit 2 (2020-09-30) I've opened a pull request (see #3213) with the new filtering implementation. Feedback is highly appreciated, since I still don't get any ads myself and therefore can't verify it. If you want to test this and give feedback, install it via python-pip like this (more details [in the install docs](https://streamlink.github.io/install.html#pypi-package-and-source-code)), but beware that there will be more changes on that branch. Alternatively, install it in a virtual-env, so that it doesn't interfere with your main install. ```bash pip install --user 'git+https://github.com/bastimeyer/streamlink.git@plugins/twitch/embedded-ads-2020-09-29' ``` ### Edit 1 (2020-09-29) Twitch has changed the annotated titles of the ads, and Streamlink doesn't expect this and thus can't filter them out. The issue can be fixed with a trivial change ([see post below](https://github.com/streamlink/streamlink/issues/3210#issuecomment-700949045)), but this may introduce other issues in the future. A better filtering approach is being worked on. This trivial change won't prevent embedded ads from happening and only fixes the filtering.

---

Issue triage (old stuff)

As you can read in the [changelog](https://github.com/streamlink/streamlink/blob/master/CHANGELOG.md), we have recently (re-)added changes in the latest release (1.6.0) to prevent the embedded ads on Twitch from happening. This workaround basically sets the `platform=_` request parameter when acquiring the stream URL and that made Twitch not embed pre-roll and mid-stream ads into the HLS stream. I've already said in #3173 that this probably won't last long, just like last time we had it implemented, and now here we are and it's broken again after Twitch has pushed another set of changes. Another issue seems to be that the ads themselves are now annotated differently and this breaks the current ad-filtering implementation, which exists in addition to the workaround request parameter. We have gone through multiple iterations here as well, because the way ads were embedded has changed quite a lot. **It appears though that Twitch is only rolling out new embedded ads in certain regions right now**, because it's not reproducible for me at the time (with a German IP address).

---

Checking the HLS playlist content of the new ads (DONE, thanks for the posts)

~~Because of that, I'm interested in the content of HLS playlists with embedded ads, so we can debug this. Please share a link to a [Github gist](https://gist.github.com/) post with the playlist content. You can check this by running~~ ```bash curl -s "$(streamlink --stream-url twitch.tv/CHANNEL best)" ``` The content should **NOT** look like this (notice the `EXTINF` tags and their "live" attribute): ```m3u #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:6 #EXT-X-MEDIA-SEQUENCE:1234 #EXT-X-TWITCH-ELAPSED-SECS:1337.000 #EXT-X-TWITCH-TOTAL-SECS:1338.000 #EXT-X-DATERANGE:ID="source-1234567890",CLASS="twitch-stream-source",START-DATE="2020-09-29T00:00:00.000Z",END-ON-NEXT=YES,X-TV-TWITCH-STREAM-SOURCE="live" #EXT-X-DATERANGE:ID="trigger-1234567890",CLASS="twitch-trigger",START-DATE="2020-09-29T00:00:00.000Z",END-ON-NEXT=YES,X-TV-TWITCH-TRIGGER-URL="https://video-weaver.XXXXX.hls.ttvnw.net/trigger/some-long-base64-string" #EXT-X-PROGRAM-DATE-TIME:2020-09-29T00:00:00.000Z #EXTINF:2.000,live https://video-edge-XXXXXX.XXXXX.abs.hls.ttvnw.net/v1/segment/some-long-base64-string.ts [... list of remaining segments and program date metadata] #EXT-X-TWITCH-PREFETCH:https://video-edge-XXXXXX.XXXXX.abs.hls.ttvnw.net/v1/segment/some-long-base64-string.ts #EXT-X-TWITCH-PREFETCH:https://video-edge-XXXXXX.XXXXX.abs.hls.ttvnw.net/v1/segment/some-long-base64-string.ts ```

[gravyboat]

On windows 10 you need to directly invoke curl.exe in the command otherwise it will run the alias. Your command should look like this: curl.exe -s "$(streamlink --stream-url twitch.tv/CHANNEL best)"

[ghost]

https://gist.github.com/fronkr56/75c54414a5aef583bf2326c914e4e1df mine looks slightly different

[bastimeyer]

These are all playlists with regular "live" content. As said, see the #EXTINF tags and their title. Also, a playlist with ads must include tags like #EXT-X-DISCONTINUITY.

[drmthrowaway]

Note that I could not reproduce unless I set Firefox headers. curl -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0" -s "$(streamlink --stream-url twitch.tv/$CHANNEL best)"

https://gist.github.com/drmthrowaway/99ddc2308ba431e76945e456da837b75

[bastimeyer]

@drmthrowaway Thank you. I already suspected it, because it's one of the differences between the HTTP requests which curl (without params) and Streamlink sends (Streamlink sets its UA to Firefox by default).

The TwitchM3U8Parser currently only finds ads which are annotated with Amazon at the beginning of the segment title. Changing this to != "live" should fix the filtering.

Completely preventing ads however is a different topic.

[bastimeyer]

Btw, I don't know if title and title != "live" is the correct solution here, because I'm not sure if this could cause issues with VODs and streams in the future. The VODs I've checked (I have no experience with Twitch's VODs because Streamlink doesn't work well with non-live content and I'm usually not interested in it) currently have empty segment titles, but if Twitch decides to change this, or if I've missed something here, or if regular live streams will have a different annotation in the future for whatever reason, this will break and everything will get detected as an ad incorrectly.

A better fix would be checking for an existing EXT-X-DATERANGE tag with an ID attribute value beginning with stitched-ad and getting its START-DATE and DURATION attribute values. These attribute values can then be used for properly filtering out segments without reading their titles, because each segment is always annotated with a timestamp and duration.

[tp0]

https://gist.github.com/tp0/f4e6798301fb8361c192e17734a0ca16

[quocka]

Any guess why I would not be seeing the pre-roll ads on 1.4? I do get them on 1.6.

[pflanzenandi]

Any guess why I would not be seeing the pre-roll ads on 1.4? I do get them on 1.6.

I am on 1.5 and I do not even have to wait for the pre-roll ads to finish anymore. Each stream starts immediately.

[bastimeyer]

As usual, Twitch is enabling and disabling embedded ads temporarily for specific events, for testing purposes or whatever their plans are. And that often times regionally, which means not everybody will get embedded ads all the time. 1.4.0 had the same filtering logic as 1.6.0, so there's no difference here other than the request parameter I mentioned in the issue post which prevents ads (unless it doesn't).

Regarding the better filtering implementation, I have done some work on my local repo and it's working fine so far, but I'm not ready to open a pull request yet. Also because of another PR that needs to be merged first which this work relies on.

[mlvzk]

Is there a diff/patch available for this that I can try?

[bastimeyer]

As mentioned above, for now, it's just that:

diff --git a/src/streamlink/plugins/twitch.py b/src/streamlink/plugins/twitch.py
index 6bdfd452..e17f0475 100644
--- a/src/streamlink/plugins/twitch.py
+++ b/src/streamlink/plugins/twitch.py
@@ -128,7 +128,7 @@ LOW_LATENCY_MAX_LIVE_EDGE = 2
 class TwitchM3U8Parser(M3U8Parser):
     def parse_extinf(self, value):
         duration, title = super(TwitchM3U8Parser, self).parse_extinf(value)
-        if title and str(title).startswith("Amazon"):
+        if title and title != "live":
             self.state["ad"] = True

         return duration, title

[expiation]

changing the above ^ does not stop ads for me

[bastimeyer]

Please read the thread. This doesn't prevent/stop ads, this fixes the filtering when having --twitch-disable-ads set.

[h1z1]

@bastimeyer - I get the feeling Twitch is reading very carfully what people are trying fwiw. The are still using that name or are rotating it.

[Terra-B-Welch]

So I have noticed inconsistencies in the ad skipping, sometimes it skips, sometimes it doesn't. I've been using the same code every time.

"streamlink --loglevel debug --twitch-disable-ads twitch.tv/(insertnamehere) 360p"

EDIT: Never mind it straight up no longer works. What a bummer.

[bastimeyer]

Btw, there's a PR open with the updated filtering logic and I would appreciate it if I could get some more feedback from users getting ads on Twitch, because I don't and can't verify that it's working correctly 100%. https://github.com/streamlink/streamlink/pull/3213 See the updated OP from yesterday for install instructions, in case you don't know.

And just as a reminder, this doesn't prevent ads, and it is still unknown if there's another workaround for that.

[drmthrowaway]

It's been working well on my end - the pre-roll ads have been correctly identified and stream starts when they're over. Fortunately (or unfortunately), over the past day and a half, I've only gotten ONE mid-roll ad, which did get filtered correctly.

[h1z1]

FWIW in the last hour or so Twitch has gone full retard wrt ads. Seeing 2minute + prerolls and mid. So while you may not be watching an ad, your player will likely appear hung and eventually timeout.

Was trying to watch the rocket launch on NASA whom have a channel on Twitch. Thankfully they also have other platforms.

[WinkelCode]

Embedded ads should be out for you as well now, just a couple hours ago ads started embedding for me in Germany.

Edit: Using the new uBlock Origin Development version works for me in Chrome though, maybe their fix can be used here as well?

[bastimeyer]

@martinsstuff This looks like a back-and-forth battle between setting the platform=_ stream token request parameter and removing it. We've already done it once, and removing it a second time is just ridiculous. https://github.com/gorhill/uBlock/commit/75c58ec7af969141296371aca2b91bea9ae91a6a#diff-d7616deaf3a39e3f19f8d70d098bfe7aR1337

It seems to work though when I test it on shroud's stream. With platform=_ => pre-roll, without, no ads.

This requires a decision how/when this parameter should be applied. We can't just add and remove it every couple of days/weeks and play a game of cat and mouse with Twitch.

---

For now, this is the diff to remove the parameter once again:

diff --git a/src/streamlink/plugins/twitch.py b/src/streamlink/plugins/twitch.py
index 6bdfd452..a327d405 100644
--- a/src/streamlink/plugins/twitch.py
+++ b/src/streamlink/plugins/twitch.py
@@ -317,7 +317,7 @@ class TwitchAPI(object):
     # Private API calls

     def access_token(self, endpoint, asset, **params):
-        return self.call("/api/{0}/{1}/access_token".format(endpoint, asset), private=True, **dict(platform="_", **params))
+        return self.call("/api/{0}/{1}/access_token".format(endpoint, asset), private=True, **params)

     def hosted_channel(self, **params):
         return self.call_subdomain("tmi", "/hosts", format="", **params)

[gravyboat]

@bastimeyer What if we simply define a second access token, one with platform="_" and one without, then we thread the plugin and whichever one doesn't get hit with pre-roll ads or an error is the one that we use?

[bastimeyer]

That doesn't solve the problem at all, because Twitch can simply make arbitrary changes to their backend and serve embedded ads no matter what parameter+value combination is set or is not set. We don't know why the platform=_parameter currently en-/disables embedded ads and why it changes over time. What's for sure though is that _ is just a placeholder and that the value is meant to differentiate between (official) apps on various platforms because of certain policies, contracts, etc. This system can change any time. And Twitch will also always be able to quickly block workarounds which ad blockers or tools like Streamlink implement.

Also, making multiple requests and loading multiple HLS streams, which first need to be resolved from the master playlists and then parsed and analyzed for embedded ads at the beginning, just slows down the initial stream loading by a lot. That nonsense is not worth the time implementing it, not even remotely close.

What we should do here is not hardcode it, because of the rapid changes, as everyone should know by now. Only applying the request parameter to the --twitch-disable-ads Streamlink parameter however doesn't make sense, as the --twitch-disable-ads parameter is meant for changing the HLS logic and it should always be possible to set it without affecting the access token acquirement logic. So what are the options here? Add another Streamlink parameter? Parse the request parameters from an env var? I don't know. Or we keep changing the parameter every time Twitch makes new changes and then publish a new release afterwards. Any volunteers for this work? Because I am already annoyed.

[gravyboat]

Fair points. I commented over in #3220.

[h1z1]

Twitch is not as smart as they think they are and I'll leave it at that. Take the target off your back. It could be argued streamlink is doing what it should and leave the implementation of ad handling to another project. Presumably the one embedding streamlink.

The eventual end goal of this will be full drm by them.

[superfreeman1989]

Any possible solution to the ads issue in the foreseeable future?

[fragande]

Is there any player out there that can handle the new ad segment filtering? I've tried latest MPV, MPC-HC and VLC and they all freeze and won't resume (or resume after a very long time (couple of minutes) but out of sync / stuttering badly) after the following line:

[stream.hls_filtered][info] Filtering out segments and pausing stream output

Streamlink is resuming Reloading playlist, Adding segment x to queue etc. as normal without errors, but none of the players above seem to be able to resume playback.

[bastimeyer]

What are you talking about? That is the whole point. This is embedded ad filtering, and not an embedded ads workaround, because there currently is none. Read the thread before posting.

If your player has any kind of synchronization issues after the playback resumes, then it's because of the stream discontinuity and the offset in stream timestamps. This can't be avoided, as Twitch simply hasn't served you the necessary data, either when watching an ad, or when filtering it out. Use a player that can handle this properly, like MPV for example. There is nothing that can be done here and I'm sick of repeating myself. If Twitch is embedding ads into the HLS stream, then you'll have to deal with it by either watching it, or filtering it out. And since your player is reading a progressive stream from Streamlink's output, watching ads is more prone to errors, since it's a different stream with different encoding parameters and will thus cause more issues with the player's demuxer and decoder than a simple timestamp offset when filtering out ads.

[fragande]

If it wasn't clear I do understand that it's filtering and not a workaround. I'm also aware that the issue isn't with streamlink but the player, but was simply wondering if there was a known working player with this solution. Neither latest MPV git build or stable version resumes playback (on my end anyway) after the filtering pause, it hard freezes and the process has to be killed manually. But if MPV is supposed to be able to handle it properly I'll look into it, thank you.

My apologies, clearly this wasn't the place to ask.

EDIT:

For anyone else having the same issue and reading this: disabling hardware decoding for MPV seems to have worked for me, but YRMW.

[bastimeyer]

Apparently, ads can be prevented again by adding player_type=frontpage to the access token request parameters, as shown here: https://gist.githubusercontent.com/odensc/764900d841cbdd8aa400796001e189f1/raw/ads.js

That's the current diff if you want to try it out:

diff --git a/src/streamlink/plugins/twitch.py b/src/streamlink/plugins/twitch.py
index b534b003..d4bf53ee 100644
--- a/src/streamlink/plugins/twitch.py
+++ b/src/streamlink/plugins/twitch.py
@@ -297,7 +297,7 @@ class TwitchAPI(object):
                 validate.get("sig"),
                 validate.get("token")
             ))
-        ))
+        ), player_type="frontpage")

     def token(self, tokenstr):
         return parse_json(tokenstr, schema=validate.Schema(

It's the same situation as the platform=_ request parameter, and it might get changed quickly again by Twitch if they see it being (ab)used by lots of users, so I don't know if it makes sense hardcoding such a workaround again and having to remove it later on because it's broken and causes other issues.

[bastimeyer]

The header post has been updated again...

[myhowbou]

I would love to edit the hard code, but How do I access the U Block Orgin script to add that line of code?

[myhowbou]

Nvm I set my userResourcesLocation to https://gist.githubusercontent.com/odensc/764900d841cbdd8aa400796001e189f1/raw/ads.js and that seems to work for now. Not sure sometimes twitch ads disappear for a bit and then come back. But if Twitch changes the parameter again how can I check to see what that new parameter is so I can add it?

[gravyboat]

@bastimeyer I think we're a bit blocked by the remaining Python 3 only stuff, otherwise I'd say we should just do another full release to try and head this off.

[myhowbou]

Nvm it didnt work lol

[myhowbou]

hmm well it only works half the time. Not sure why it's inconsistent.

[bastimeyer]

@myhowbou can you stop spamming this thread with unrelated posts? This is about Streamlink, not about any browser related ad blocking stuff. The request parameter is working fine here, and if it doesn't for you in another region or so, then be absolutely clear about that with an example log post. I'm tempted to mark this as offtopic, because I really don't like bloat in such megathreads, and everyone seeing nonsense notifications probably neither.

@gravyboat What do you mean? 2.0.0 is technically ready, as every mandatory change has been merged. Everything else listed in the checklist is optional. I don't see the need of rushing another release right now though just because of this change. The workaround is very simple, and I have provided a diff and also instructions for installing the latest development version, and that should be good for now until we're all confident with publishing the 2.0.0 release in a good shape. Making another 1.x release would mean that we'd have to branch off from an earlier commit, which doesn't make sense considering that we explicitly chose not to do the py2 removal on a secondary branch (now the situation is in reverse). If that's a problem now, then we should have switched to a different git flow instead of having just one main branch.

[myhowbou]

@bastimeyer Hmm. Okay sorry. I thought this was a patch for the new twitch thing for Ublock.

[ghost]

I did a pip upgrade and suddenly twitch displays those blank 10 sec moments for ads again? What happened?

[gravyboat]

@bastimeyer Sorry I should have been clear, I agree with you that the workaround is simple but it was simple last time too when people needed to install the nightly and we still had tons of comments. I do not want to make another 1.x release.

[bastimeyer]

@tarkov2213 You have either not installed the latest dev version, or you are using an older version in parallel by accident. As I've said in my previous comment, one should post a debug log (with --twitch-disable-ads set, so it'll include logs for ad segments) if they think that it's not working correctly, because non-meaningful statements like this without any information at all are a waste of time for every developer, regardless the project.

---

@gravyboat If there's going to be spam or more repetitive questions of people who haven't read anything before posting, then we're going to lock it. This should not make us rush a new release, because there are some changes left which I think should be done/finished first. I hope that's somewhat understandable.

So far, it's been rather tame though. But I also want to mention that due to being the Twitch GUI dev, I've been answering questions left and right each day on various other channels as well, and this gets very annoying over time, especially since I don't get paid for "playing unofficial Twitch support staff" while working on an unrelated project, so I understand that you want to push this and be done with it. But remember that the workaround can stop working at any moment, and if we rush things because of it, it might backfire.

Once again, the current solution for the enduser is very simple, and if some people are unwilling to read or don't understand, or just want to complain, then it's not my problem.

[S0und]

Offtopic:

Interesting, they've moved the HLS request from api.twitch.tv behind graphql

https://i.imgur.com/DteEeoC.png

[arrowgent]

i put this twitch.py patch and seems to work fine (linux) no need to push a release

cat & mouse either way btw, the "commercial break" screen is satisfactory...atleast its not an actual ad thanks for your work and that "frontpage" discovery lol

[Darushin]

Just wanted to add some odd behavior I am observing. I am using the precompiled windows binaries for twitch (latest stable) and am noticing for channels I am subscribed to I am seeing ads. This happens regardless of tier level (I have tier 1-3 subs) and I verified the ads are not being ran by the streamers. I patched in the python above and I am seeing ads for streamers I am subscribed to still (before I was seeing the commercial break screen). I checked streamers that I am not subscribed to and saw ads randomly (5 saw ads 1 did not). If there is any logs you would like me to provide just let me know what you need and how to produce them and I will send them.

[bastimeyer]

@S0und I don't see anything about that on Twitch. They are still using the old private API for the access token. If they had rolled out those changes, then we'd seen tons of posts here, because it'd break the Twitch plugin. It is possible though that this change will come at some point.

---

@Darushin And how exactly do you think Streamlink knows about your subscriptions? Any kind of authentication has been removed from previous Streamlink versions (see https://github.com/streamlink/streamlink/issues/2680#issuecomment-557605851). https://github.com/streamlink/streamlink/blob/master/CHANGELOG.md Also, the last stable release is pretty much offtopic at this point since #3301 got merged.

[ghost]

@bastimeyer well excuse me for assuming the package you have in the pip repository would contain important fixes like no ads. uninstalled it and pip installed straight from git+github and the problem is fixed.

[bastimeyer]

If this change was included in the latest stable release, then we wouldn't be discussing it here.

---

I've just noticed that the new ads prevention request parameter is not working on every Twitch channel, unfortunately. It's working fine on most, but on esl_dota2 for example, you'll still see preroll ads. Not sure why.

[bastimeyer]

In addition to player_type=frontpage, it seems that player_type=dashboard is another way of preventing embedded ads.

[S0und]

I had to switch from frontpage to embed yesterday. Seemingly worked for a lot of people, but someone reported ads today even with that.

https://gist.githubusercontent.com/odensc/764900d841cbdd8aa400796001e189f1/raw/ads.js

dashboard is also a variant, but i haven't tried it yet.

[pixeltris]

When mid-roll ads play a mini-player is created so that the viewer can still see the stream whilst ads play. Whilst it's limited to 360p it might be an idea to use this in cases where ads cannot be blocked by streamlink. There's some additional discussion on that here https://gist.github.com/odensc/764900d841cbdd8aa400796001e189f1#gistcomment-3510566