General discussion

[ghost]

Hi Pixeltris,

Just wanted to let you know, I have made some changes to 2.4 of Video Ad Block, I found a way to add in a check for a source quality ad-free stream. It seems to work quite well, sometimes a tiny buffer happens but it reduces the time of the 480p res significantly and on some streams it's 1080p straight away, others, it's like 10 or 15 seconds. The only downside is the amount of gql requests made, but it doesn't seem to have any downsides to the user experience..

The content.js https://github.com/saucettv/VideoAdBlockForTwitch/blob/main/content.js

You are obviously, very welcome to adapt it for notify-strip.

I also managed to finally test mid-rolls and it was indeed the picture-by-picture request causing the play/pause loop, so making the gql request for it empty fixed that issue and mid-rolls change res seamless also.

Cheers

[pixeltris]

Yea I could see that helping where the server accepts the "ad watched" notification, then the site player type subsequently returns a stream without ads.

Another interesting idea (which I haven't actually tried, but should in theory work) is to use a static Device-ID / X-Device-Id. As the "ad watched" notification is directly tied to the device id, using a shared value would greatly reduce prerolls for all users. A generative rolling algorithm which generates a device id on a daily, or weekly bases would probably be better than a pure static value though, as Twitch could just flag any such static device id.

[ghost]

That's a good idea, will give it some testing and see if it works, not sure if Twitch verifies if a device ID exists or not, but will find out. I'm thinking maybe use day of the month as an index and then change the two numbers/letters according to that, so that each day will be a unique device ID, in a different order. That should also prevent Twitch from blocking a range of them. I'm also looking into other ways of notifying, as I have a feeling using the impression complete request will be a bad idea for the future as it technically causes Twitch to give advertisers false stats (may cause them legal issues in the future). I know they have an ad declined request due to rate limit, might see if that works instead.

[ghost]

Found some interesting info in their settings.js file, especially in the exp section. Particularly, "lol_extension_installation", could mean league of legends, but could also mean the ad-blocker, as it's just above an ad related exp. Found a few other exp related to the player, so they could already be trying ways to circumvent the content script. Looks like they might be about the to try delayed pre-rolls too. So when doing the notify, we will need to include "POSTROLL" as well as MIDROLL AND PREROLL. It might also include time_break in the gql request. Won't know until they try it though.

[pixeltris]

lol_extension_installation has been around for some time (at least 5 months based on this Twitch tracker).

I played around with the decline ad request some time ago, but I couldn't get any results with it. Maybe I was doing it wrong, but it could just be more of a analytics thing.

Delayed prerolls sounds interesting, that might actually be an avenue for reducing ads by exploiting it if that becomes a thing.

[ghost]

I've also been looking into the Channel_SetSessionStatus request. Seems it calls it after the usher request with around a 400 second countdown. Then on page refresh it calls a command to drop the context of it. I wonder if calling this method before the usher, would set a ad-free cooldown for the sessionid. As it seems, this might be another way of them implementing a pre-roll rate-limit. When the 400 seconds runs out, it can call an ad, but if we were to call the request before each usher request, the cooldown would always still be running. Haven't tested it though.

[ghost]

Another small bit of information I found out. In the usher request, server_ads=true means they will show embedded stream ads. I noticed one day that all my usher requests had this set to false while testing the extension. So I forced a new device id and then all of a sudden server_ads was true again. So the embedded ads can also be enabled/disabled by device ID. I just don't know what turns it on/off. It also means there may be a whitelist for a certain device ID to not get server ads.

[pixeltris]

Yea, the "server_ads":false was always a good indication of finding a bypass for ads. This was observed from the initial push Twitch made towards SSAI https://github.com/streamlink/streamlink/issues/2357#issuecomment-473565335 and many workarounds since then which have been patched. I'm sure there are still many similar workarounds.

I've observed device ids which never get ads as well, not 100% sure what triggers that exactly.

FYI I've been using Video Ad-Block, for Twitch. Inside the reloadTwitchPlayer function one of those objects (player/playerState) holds the currently set resolution (from player settings) which might be useful to make the "source' resolution part go to the currently set resolution (some minor m3u8 parsing required). I see you've been making some changes, but exact resolution might be useful to some users. I've also noticed that during midrolls sometimes (rarely) one of the m3u8 requests fail (I think the urlInfo request), as they aren't try/catched the stream freezes until manually fixed. But otherwise working great.

[ghost]

Oh that's interesting, thanks for letting me know :)

[ghost]

I found that using player.seekTo(0); instead of player.pause(); then player.play();, works much better. You get about 500ms of buffer but no other UI effects. Actually it does sometimes still show the controls, but you don't see the big play icon in the middle.

[pixeltris]

Yea I went for seekTo to avoid the play icon. Also I assume that issue people were having with tabbed windows pausing was a result of the pause/play? So I wanted to avoid complications with that.

The seekTo might actually have other side effects so pause/play might be better. And as you mention the seekTo has worse time shifting compared to pause/play which might be a big deal for users.

[ghost]

I think the next thing to search for is a way to apply the rate limit, without notifying an impression. I've been looking at lots of ways, but haven't found anything yet. It's actually a bit a dilemma when they end up coming to try and circumvent these methods, because they would have to remove features that they seem to rely on.

[ghost]

I ended up changing the baseData of the notify request to

            stitched: true,
            roll_type: rollType,
            player_mute: true,
            player_volume: 0.0,
            visible: false,

It still applies the rate limit, but should prevent advertisers from getting false impression data, because I guess an ad that is muted and not visible, isn't really an impression.

[pixeltris]

Hard to tell what impact that would actually have, but probably a good idea. It's really the fault of Twitch that such request is necessary.

Raids might be worth looking at. As far as I know raids never get ads. However, I think you need a valid raid id. There's also secondary players in squad mode which get source resolution without ads. But again I think you actually need the stream to be in squad mode. I haven't fully investigated these but they are worth exploring.

[pixeltris]

seekTo(0) can reverse time after the ad finishes, I think this happens in cases where time "0" is actually in the buffer. Switching to pause/play for now. I wonder if that browser tab visibility negation has any negative performance impacts.

[ghost]

Oh that's annoying. Yeah I'm not sure, it shouldn't for users who only have one tab open. Pretty sure multi tabs still all have the chat still running etc, so should be minimal if it does. I'm not sure if I tested all of the things I blocked, I know the visibilitychange one is the main one, but I had to add another and I can't remember the reason why now.

[ghost]

I was thinking of trying to hide the player controller as the script pauses/plays and then show it again after, to make it seem like a more seamless exp. Been playing around with it today. Got it to hide the controller, but as soon as I remove the hidden style element, it automatically shows itself anyway. I'm wondering if there is anything in the mediaPlayerInstance that can disable the controller.

[ghost]

I did manage to make it basically seamless when it transitions from the ad blocking to the normal stream, but with one issue, I had to hide the controls for 6 seconds after returning to a normal stream. Whether this would be noticed by the user or not, I'm not sure. This is what I did though:

function reloadTwitchPlayer(isPausePlay) {
    //This will do an instant pause/play to return to original quality once the ad is finished.
    document.querySelector('.video-player__overlay').style.visibility="hidden";
    function findReactNode(root, constraint) {
        if (root.stateNode && constraint(root.stateNode)) {
            return root.stateNode;
        }
        let node = root.child;
        while (node) {
            const result = findReactNode(node, constraint);
            if (result) {
                return result;
            }
            node = node.sibling;
        }
        return null;
    }
    var reactRootNode = null;
    var rootNode = document.querySelector('#root');
    if (rootNode && rootNode._reactRootContainer && rootNode._reactRootContainer._internalRoot && rootNode._reactRootContainer._internalRoot.current) {
        reactRootNode = rootNode._reactRootContainer._internalRoot.current;
    }
    if (!reactRootNode) {
        return;
    }
    var player = findReactNode(reactRootNode, node => node.setPlayerActive && node.props && node.props.mediaPlayerInstance);
    player = player && player.props && player.props.mediaPlayerInstance ? player.props.mediaPlayerInstance : null;
    var playerState = findReactNode(reactRootNode, node => node.setSrc && node.setInitialPlaybackSettings);
    if (!player) {
        return;
    }
    if (!playerState) {
        return;
    }
    if (player.paused) {
        return;
    }
    if (isPausePlay) {
        player.pause();
        player.play();
        setTimeout(function() {
        document.querySelector('.video-player__overlay').style.visibility="visible";
        }, 6000);
        return;
    }
}

[pixeltris]

Nice. I think I'm going to stick with the visual artifacts for the time being.

You might want to put the hidden just before the pause/play calls as there could be a race condition of sorts on that player.paused check which could potentially hide the controls indefinitely.

[ghost]

Oh yeah, good spot, thanks

[ghost]

Had a play with a random device ID code. I made something that creates a daily random ID by using a random letter/number, but also a random location to change it, it will change daily. There seems to be only one drawback and that is day 1 and 2 will have the same device ID. I tested using currentDayNumber as 1 and also 365 and it passes ok. Let me know if you see anything wrong.

var randomDeviceID = 'eVI6jx47kJvCFfFowK86eVI6jx47kJvC'; var alphaNum = 'abcdefghijklmnopqrstuvwxyz123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567'

var currentDate = new Date(); var currentDateStart = new Date(currentDate.getFullYear(), 0, 0); var dateDifference = (currentDate - currentDateStart) + ((currentDateStart.getTimezoneOffset() - currentDate.getTimezoneOffset()) 60 1000); var oneDay = 1000 60 60 * 24;

var currentDayNumber = Math.floor(dateDifference / oneDay); var letterNumLocation = currentDayNumber / 6; letterNumLocation = letterNumLocation.toFixed(0); var letterNumToChangeTo = alphaNum.charAt(letterNumLocation); letterNumLocation = letterNumLocation / 2; letterNumLocation = letterNumLocation.toFixed(0); randomDeviceID = randomDeviceID.split(''); randomDeviceID[letterNumLocation] = letterNumToChangeTo; randomDeviceID = randomDeviceID.join('');

console.log('Location of letter or number to change: ' + letterNumLocation); console.log('Letter or number to use: ' + letterNumToChangeTo); console.log('New Device ID: ' + randomDeviceID);

[pixeltris]

Looks good, using UTC there is a good idea and something I should have probably done. I'm going to stick with the algorithm in this repo for now as the limited number of users have stopped all prerolls as far as I can tell. I feel this will most likely get patched at some point due to this fact.

So many people are running midrolls these days. Looks like there's a fair few issues with notify-strip with midrolls. I need to spend time improving it.

[ghost]

I have just uploaded new versions of the extension 4.2 that includes the random device ID and seamless transition, so will see how it goes.

[pixeltris]

It looks like there's now an IP check on the device id. This still works in a limited capacity (i.e. within a LAN setting).

I'll disable device id sharing soon as it potentially gives Twitch the ability to do unwelcome things.

EDIT: Actually it looks like they are force serving ads on every page load with the shared device id. Maybe they are detecting these specific ones, or ones not provided by Twitch themselves. Pretty shitty thing to do though. So yea I'll go ahead and disable that soon. (or they are just being aggressive with with prerolls, not 100% sure.)

[ghost]

It could be that the notify function needs a real device ID for it to work and kick in the rate-limit.

[ghost]

I noticed the video_ad_pod_complete on it's own is not enough now. It needs all the quartiles too.

[pixeltris]

Oh right that might be part of the issue. I'll have to play round with that.

[ghost]

Twitch is currently trying to manipulate me into revealing to them if I am earning money from the extension via email. It's not an email address directly from Twitch, I'm guessing they are using some law firm or private investigator to gather evidence. It's fairly obvious to me, as I have some legal knowledge via family members. Just a warning, that they are very aware of the scripts. The scripts, as far as a know, are not breaking any laws, we are in fact using their own player types at the end of the day. We are also not blocking any paywalls etc, so I guess they are trying this route first. The ads are random, the live copyright is on the streamers live stream, the ads are added in on a random basis (without the streamer knowing the ad's content) to each user, so no user is guaranteed the same viewing experience. We also notify Twitch that the ad was not visible. On this basis, I don't think Twitch could claim we are removing content that should be shown to all users, especially when the scripts don't actually remove anything, they simply swap their own player types on a per user basis. As far as I know all law cases between ad-blockers and advertisers/publishers has been won by the ad-blockers. The unknown part is when ad-blockers bypass paywalls, which we do not.

My guess is they will next implement something to stop the rate-limit and eventually remove the pbyp player and put their ad serving scripts inside a code that we can't block directly, that's if the next CEO decides to continue what the last one has been trying to do. I really do hope they realize the mistakes in pushing advertising this hard, especially when the typical ad-block user is a fraction of their total user-base. I have no doubt their viewer numbers would of been higher than they are now, if not for the ads. The fact that they now have millions of affiliate streamers with 0 viewers, in an attempt to bring in more ad revenue, shows me that the price of their bandwidth is not as much of a problem as people assume.

Amazon are pushing hard for their own advertising platform, a bit like MoPub, AdSense etc, so I'm guessing the pressure is high. They have already implemented it into their overlay type ads via the "ProgrammaticAdRequesting" GQL request. So we should keep an eye out for any changes that may come for those types of ads.

I have already noticed a decrease in stitched based ads, compared to a few months ago. Advertisers, are also taking notice.

There are GQL requests, that send data about a users ad identity, this could be blocked fairly easily by checking the GQL body. I'm unsure if this affects only the overlay type ads or the premium stitched based ads also. This could also be done server-side and we should keep an eye out for it.

[ghost]

Also noticed a new header for the GQL request for the token, Client-Version: ****

[pixeltris]

Interesting. They haven't made any real technical changes in a while (other than minor tweaks). It'll be interesting to see if that changes.

Hopefully the legal stuff is just some unaffiliated disgruntled person pretending to be an authority.

[ghost]

Yeah hopefully, I'm not too bothered because if Google and Mozilla keep it up and distribute it, then I very much doubt they have a case.

I think they think they closed the case when they changed the embed player. They made out that it somehow changed the actual player on the page, when we both know, it was nothing to do with the player itself, just the stream that came through. I think there were some Twitch devs that were unhappy about that decision too. Other than going back on themselves and removing the pbyp player or adding little obstacles, I'm not sure what else they could actually do that we wouldn't eventually be able to work around. They must also have some kind of contract with the esports companies for that player type.

I'm just suprised about the stitched ads, because there are no cookies and you can send a blank device ID and still get an ad, so the targetting for the advertisers must be really wide and advertisers seem to hate that right now. Especially with the likes of Facebook where you can target little timmy at 45 timmy street who likes to eat pancakes on a tuesday. Although they do target using IP, but it only seems to be by country, rather than them using the IP as personal information. I tested this using a VPN to do the GQL request to get the token and then another country to get the usher request and it only uses the IP sent to the usher url, not the one in the gql token to request the ad. The usher IP was new and non personal so it wouldn't of been connected to them in any way.

I'm hoping they just scrap SureStream and use normal overlay ads/sponsors etc, let the blockers block, increase viewership, bring back the community and increase revenue via that, rather than ads. SureStream would of been good tech for the likes of YouTube, because the videos are not live, but the whole pre-roll, mid-roll thing handled by the streamer is what is messing both parties up. Advertising in my opinion is the worst way to see how a business is performing, pushing more ads will of course increase revenue in the short term (Makes a CEO look good just before they leave), but Subs, bits, donations etc would of been a better push to see how the platform is doing and they pushed that down big time. Now they have a silly amount of 0 viewer streamers that can't get views because of how off-putting it is and it's only increasing, including the bandwidth. Madness to me.