Problem with generating new cookies file

[XaaRii]

Describe the bug Problem with generating new cookies file, but existing ones work just fine

To Reproduce Steps to reproduce the behavior:

1. Download latest version (currently 1.3.4)
2. Install prerequisities with pip install -r requirements.txt
3. Copy the example file and change the username (password set to "" -> will ask interactively)
4. Run the modified script, be asked to enter the password
5. After you fill the password, it crashes.

Expected behavior Expected a question for 2FA and the Twitch Miner to start.

Desktop (please complete the following information):

• OS: Canonical Ubuntu with architecture aarch64 (arm64) (you could think of it as running on Raspberry Pi, never had problems tho)
• Python version 3.10.6

Additional context Problem with generating cookies file, because it works if i provide them one generated from before (i believe it was a version 1.2), but sadly, this problem occurs minimally from version 1.3.2 (tested this version, then moved on to the latest and the problem was still present). Would be amazing if that could be fixed.

Log:

28/10/22 18:59:34 - INFO - [run]: [username] Start session: 'sessionid' 28/10/22 18:59:34 - INFO - [login_flow]: [username] You'll have to login to Twitch! Enter Twitch password for username: Traceback (most recent call last): File "/home/ubuntu/twitchminers/runPawele.py", line 110, in twitch_miner.mine( File "/home/ubuntu/twitchminers/TwitchChannelPointsMiner/TwitchChannelPointsMiner.py", line 161, in mine self.run(streamers=streamers, blacklist=blacklist, followers=followers) File "/home/ubuntu/twitchminers/TwitchChannelPointsMiner/TwitchChannelPointsMiner.py", line 179, in run self.twitch.login() File "/home/ubuntu/twitchminers/TwitchChannelPointsMiner/classes/Twitch.py", line 84, in login if self.twitch_login.login_flow(): File "/home/ubuntu/twitchminers/TwitchChannelPointsMiner/classes/TwitchLogin.py", line 75, in login_flow login_response = self.send_login_request(post_data) File "/home/ubuntu/twitchminers/TwitchChannelPointsMiner/classes/TwitchLogin.py", line 152, in send_login_request return response.json() File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3.10/json/init.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 340, in decode raise JSONDecodeError("Extra data", s, end) json.decoder.JSONDecodeError: Extra data: line 1 column 128 (char 127)

[PingkeuKeobi]

I can confirm, I have the same issue.

[MyNameJeff12123]

Twitch have now changed the Login, the client no longer works. {"error":"Please update your app to continue","error_code":5023,"error_description":"client is not supported for this feature"} Simply changing the client ID is not enough, I believe they now require a Integrity Check too.

[Rakambda]

I tried with the iOS app, not android, but I believe they work the same for login. And indeed they now have the integrity token and login call is now on /protected_login.

[MyNameJeff12123]

I don’t think it does. I think it works a little different because you cannot send oauth without having logged in? Sent from my iPhone 13 Pro MaxOn 29 Oct 2022, at 09:11, Thomas @.***> wrote: I tried with the iOS app, not android, but I believe they work the same for login. And indeed they now have the integrity token and login calls is now on /protected_login.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[Mohamed6935]

So any idea how to fix this?/workaround

[MyNameJeff12123]

So any idea how to fix this?/workaround

I believe the dev is aware, and probably working on the authentication system, I also am looking into this.

[Mohamed6935]

Last question since not sure if it is related to this.i am currently running the pc version (not docker) and for some reason I only got raid points and no watch/claim . Even though the day before I could half them all even with 2 streams running. Nvm just did a test and it works so maybe it didn't work for the last night stream because I didn't have a verified phone and it needed it

[rdavydov]

If send_login_request to https://passport.twitch.tv/protected_login has no integrity_token:

Unknown error: {'error': 'Oops! We encountered an unexpected error. Please try again.', 'error_code': 5022, 'error_description': 'missing integrity token'}

This will require some severe changes to TwitchLogin.py, now it also needs to get integrity_token. And we need to figure out what GQLOperation will it be - ReportMenuItem or something new. 🤔

@bossoq please take a look and help us if you have time.

[MyNameJeff12123]

If send_login_request to https://passport.twitch.tv/protected_login has no integrity_token:

Unknown error: {'error': 'Oops! We encountered an unexpected error. Please try again.', 'error_code': 5022, 'error_description': 'missing integrity token'}

This will require some severe changes to TwitchLogin.py, now it also needs to get integrity_token. And we need to figure out what GQLOperation will it be - ReportMenuItem or something new. 🤔

@bossoq please take a look and help us if you have time.

Even on desktop browser as a phone I get HTTP 400 Error / {"error":"Oops! We encountered an unexpected error. Please try again.","errors":["Oops! We encountered an unexpected error. Please try again."],"error_code":5025,"error_description":"integrity failed"}

[rdavydov]

@MyNameJeff12123 I didn't ask. Maybe now he will join, since you mentioned him. ;)

I'll post my research log here:

Requests made by the Twitch Android app after you tap "Log In":

• POST to https://passport.twitch.tv/integrity - nothing new in headers - returns { "token": "<TOKEN>" } in RESPONSE BODY
• POST to https://passport.twitch.tv/protected_login - nothing new in headers - in REQUEST BODY "integrity_token": "<TOKEN>" that we got above

[Rakambda]

I think integrity also fills up some cookies. Basically it's the same flow as with the web version.

[rdavydov]

@Rakambda Yeah, I think you mean these:

set-cookie:
ga__12_abel-ssn=<OMITTED>; Max-Age=86400; Path=/; Expires=Sun, 30 Oct 2022 13:44:35 GMT; HttpOnly; Secure; SameSite=None

set-cookie:
ga__12_abel=<OMITTED>l; Max-Age=86400; Path=/; Expires=Sun, 30 Oct 2022 13:44:35 GMT; HttpOnly

[MyNameJeff12123]

Twitch are using Kasada to protect against bots. @6accOnThe6locc hopefully will help with automation

[Rakambda]

Yeah and kasada is annoying. Personally on my project, before adding the solution with the mobile client-id, I simply hooked it with a selenium instance pulling the integrity token regularly from the website itself.

[Mohamed6935]

Is the 2 streams limit got reduced to only one

[rdavydov]

Kasada headers are present, don't know if they were there before when we was able to just simply omit them on /integrity. If they are required now - that is a bummer.

[rdavydov]

Well, it seems that sending an empty POST request to https://passport.twitch.tv/integrity without the Kasada headers fails the integrity.

In this case our only solution is pulling the integrity token via a Selenium instance. This may take a lot of time to implement and test. No workaround for now.

[Rakambda]

If you want an inspiration by how I did it : https://github.com/Rakambda/ChannelPointsMiner/blob/develop/miner/src/main/java/fr/rakambda/channelpointsminer/miner/api/gql/integrity/browser/BrowserIntegrityProvider.java This part is for the GQL integrity.

I never managed to get the passport one working. So I just login through the browser and then import the cookies (this isn't an issue as passport is long-lived unlike the gql that needs to be refreshed somewhat daily).

[DevilXD]

@Rakambda Just wondering, did it work out long-term? My personal research tells me that Twitch will detect and deny any non-mainstream browser, other than Chrome or Firefox. Haven't tried Opera yet, but Selenium-based webdriver is definitely detected too. As well as Brave browser that I've seen users complaining about not being able to claim drops, shortly after the integrity system got implemented.

To add to @rdavydov's research, from my side:

• The integrity token is actually a urlsafe-base64-encoded JWT, containing (among other things) a boolean "is_bad_bot" flag. Getting a "true" value there is pretty much guaranteed to make Twitch error out on any GQL request at some point. Trying to "refresh" this value when that happens leads nowhere. You can find some Python code that extracts this value below. @rdavydov may have already seen it somewhere, as I saw someone they know "borrow" that piece for their testing purposes. No issue taken. • Not all GQL operations are protected by the integrity system. The only operations prone to failure appear to be ones attempting to claim things. Inventory and campaigns fetching remains unaffected. • I've had someone dive pretty deep into the webdriver-controlled browsers topic, and they've tried using a setup that can successfully login to Google, to try and obtain a non-rigged ("is_bad_bot": "false") integrity token - and they've failed. Whatever this Kasada thing is, it's pretty resilient to everything being thrown at it so far. • Merely accessing http://twitch.tv without logging in is already enough to see and test the integrity system in action. This allows for harmless experimentation with trying to obtain a non-rigged token, without providing any identifiable information, maybe besides your IP address and a browser fingerprint.

Now, as far as my understanding of the information flow goes: • Browser makes a request to the http://passport.twitch.tv/*/*/fp and http://gql.twitch.tv/*/*/fp endpoints. "fp" suggests to me it probably stands for "fingerprint", and whatever JS code is obtained from those endpoints, creates a breakdown of all features of the browser. It's meticulously checking every single nook and cranny of your browser to try and identify it by it's available features. https://k.twitchcdn.net/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/p.js is an example of one of the scripts (if not the exact thing) responsible for handling of that. Those very long hex numbers in the URL are the same for the p.js file, and fp and tl endpoints, and thus there might be multiple different copies of this p.js file, under different URLs. Cracking just this one won't crack all of the others. • The fingerprint is sent and registered by the server at the http://passport.twitch.tv/*/*/tl and http://gql.twitch.tv/*/*/tl endpoints, at which point the server determines if the sender is a bot or not, based on the fingerprint payload and all of the attributes your browser has. • In addition to this, p.js generates the values for the integrity headers - the cryptographic challenge. • The results of the challenge, together with the fingerprint information, is used by the server to generate a non-rigged token. The tl endpoint responds with just "reload: true", and I have no idea how it identifies the fingerprint -> challenge chain. Could be just IP addresses, or one of the many headers that are passed around during the exchange. • The token is used as you might expect from here.

As someone who struggles to read and understand any advanced JS code, this p.js file is a total nightmare. No amount of deobfuscators really helped me understand what's happening, yet along attempt cracking it. Knowing that there might be more, different versions of the same file, is like beating an already dead horse, really. If a "full stealth selenium custom build driver" that can beat Google, can't beat this Kasada shit, then I doubt any of us will do so. Spoofing the fingerprint and solving the challenge is like an entire 2nd project on it's own.

"is_bad_bot" flag extraction:

import re
import json
from base64 import urlsafe_b64decode

integrity_token: str = "..."

stripped_token = integrity_token.split('.')[2] + "=="
messy_json = urlsafe_b64decode(stripped_token.encode()).decode(errors="ignore")
match = re.search(r'(.+)(?<="}).+$', messy_json)
if match is None:
    raise Exception("Unable to parse the integrity token")
decoded_header = json.loads(match.group(1))
if decoded_header.get("is_bad_bot", "false") != "false":
    print("Detected")
else:
    print("Undetected")

Or just use https://base64decode.org for quick checks during in-browser dev tools testing.

Hope this helps.

[Rakambda]

Just wondering, did it work out long-term? My personal research tells me that Twitch will detect and deny any non-mainstream browser, other than Chrome or Firefox. Haven't tried Opera yet, but Selenium-based webdriver is definitely detected too. As well as Brave browser that I've seen users complaining about not being able to claim drops, shortly after the integrity system got implemented.

It's been a month that I use it and it's still working. The login part never worked, the selenium instance is somehow detected. But for the GQL part it isn't.

Not all GQL operations are protected by the integrity system. The only operations prone to failure appear to be ones attempting to claim things. Inventory and campaigns fetching remains unaffected.

Everything on the GQL is protected. What works is getting the +10 points (spade url, can be found on the webpage itself) and getting the "notifications" of what's happening on your account (pubsub api, public).

I've had someone dive pretty deep into the webdriver-controlled browsers topic, and they've tried using a setup that can successfully login to Google, to try and obtain a non-rigged ("is_bad_bot": "false") integrity token - and they've failed. Whatever this Kasada thing is, it's pretty resilient to everything being thrown at it so far.

Never managed to get the login part to work either. But once you're logged in (import cookies from normal browser), it's ok.

Now, as far as my understanding of the information flow goes:

I understood the same thing

[6accOnThe6locc]

You complicate things too much, for now I managed to do it https://github.com/6accOnThe6locc/twitch-mobile-token

Note: I found a way to circumvent but it's too risky to publicize it. Someone who understands a little can rewrite the method using selenium, I'm out of time, I can do it as soon as I have time.

[6accOnThe6locc]

I found a simple way to login to twitch with any client_id using seleniumwire https://github.com/6accOnThe6locc/twitch-mobile-token

[rdavydov]

@Rakambda @DevilXD @6accOnThe6locc Thank you guys for sharing, so much useful info already, this is what open source is all about! 🙂

@DevilXD It is interesting that previously obtained "console" cookies work with the GQL requests made by the current version of the miner. Looks like is_bad_bot becomes True when the points miner logins to Twitch with cookies extracted from the browser with browser_cookie3. I guess it is because the auth-token in those cookies doesn't "match" the mobile client_id on Twitch's side. Your is_bad_bot checking procedure is definitely worth integrating into the points miner, so it will inform the user. I guess you won't mind if we integrate it, right?

@6accOnThe6locc I was also experimenting with undetected_chromedriver before going to sleep and wanted to implement it the same way. But then I thought that many people will run it on headless instances with no display and only with a ssh access. This will be a problem, need to think about it. I saw that pyvirtualdisplay was used in this case, but it is *nix-only. There is also a --headless option for the Selenium webdriver. Should we stick with the Selenium method or maybe there is another way?

[6accOnThe6locc]

If it's just to use headless, I can finish what's left.

[rdavydov]

@6accOnThe6locc Good, I guess we have to get the verification code from the user's console input and then input it digit-by-digit the same way via driver.find_element().send_keys().

@DevilXD I've checked the integrity tokens with my old cookies obtained via the console login and they all have is_bad_bot = true. Bonus claiming still works nevertheless. I guess Twitch just lets us claim even though they are aware that we are bots. ;)

[6accOnThe6locc]

Ready guys, headless option working https://github.com/6accOnThe6locc/twitch-mobile-token

[rdavydov]

Ready guys, headless option working https://github.com/6accOnThe6locc/twitch-mobile-token

Unfortunately, with headless mode enabled, Twitch does not send a verification code. I believe they detect it and this may be our biggest problem. I've also tried geckodriver - same thing (although it works way faster than chromedriver).

[DevilXD]

@DevilXD I've checked the integrity tokens with my old cookies obtained via the console login and they all have is_bad_bot = true. Bonus claiming still works nevertheless. I guess Twitch just lets us claim even though they are aware that we are bots. ;)

Hmm, that's strange. In my testing, any rigged integrity token stopped working after a couple of hours. way before it'd normally expire, and trying to obtain a new one worked, but this new token would immediately fail as well, leading to a loop of token refreshing and failing.

Also, yeah, I don't mind those couple of lines. Use them as you see fit.

[rdavydov]

https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/tree/selenium-gql

I've made a version that gets a GQL integrity token with Selenium, using cookies extracted from the browser (had to rewrite the original code) - no luck, "failed integrity" when claiming bonuses, even though is_bad_bot = false. It becomes true when I use headless mode.

Second approach will be only logging in with Selenium, getting an auth-token and then use the old code. I hope it will be enough for now. WIP: https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/tree/selenium

Third approach will combine previous two.

And the fourth approach will be the top secret trump card - the last resort when and if everything else fails. ;)

[FarmerBrianW]

I have a workaround. From a Chrome browser, get an extension called Cookie-Editor. Log into Twitch normally, then with the Cookie Editor, you can capture the twitch cookie to your clipboard, but it is in JSON format. Next, you need to go to the cookie folder in your bot and convert that JSON to a PKL file. I wrote this little python script to do this:

import pickle
import json
import sys

for username in sys.argv[1:]:
        data = ''
        with open('cookies/' + username + '.json', 'r') as f:
                data = json.load(f)

        with open('cookies/' + username + '.pkl', 'wb') as f:
                pickle.dump(data, f)

1. save this script asjson-to-pickle.py in the same folder as your run.py bot
2. Log into your Twitch stream normally 2FA, and export the real Twitch cookie from Chrome Cookie-Editor to your clipboard
3. Create a new text file in your bot's cookie folder called your-twitch-username.json (replace your-twitch-username with your Twich username)
4. Paste the contents of your clipboard in this new file
5. Go up the directory and run python3 json-to-pickle.py your-twitch-username
6. Check the cookies folder, and you should see a your-twitch-username.pkl file
7. Run your bot with your twitch username and password hard coded Some issues, it does not seem to collect bonuses, but it does collect channel points.

[rdavydov]

@FarmerBrianW Thanks for the effort, but the current version already can do what you proposed. ;) And it is doing it automatically. We're working hard on making it able to CLAIM bonuses, etc - this is what it is all about.

[rdavydov]

https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/releases/tag/1.4.0

Second approach is working beautifully. Also, I've found what I forgot to do in the first one. Will fix it and test later.

For now, I think we can use the second approach. You will need a display, NO CAN DO on a headless machine!

First of all, install and upgrade requirements: pip install -U -r requirements.txt

• If you don't have cookies, a browser window will open, it will login with your data.
• After that, enter your verification code in the browser and wait for the Twitch website to load, then press Enter in the console.
• Everything should be the same afterwards. It collects points and claims bonuses, hooray! 🙂

[realAbitbol]

Using the latest 1.4.0 version, on MacOS Ventura, I'm getting :

31/10/22 17:22:30 - INFO - [login_flow_backup]: Enter your verification code in the browser and wait for the Twitch website to load, then press Enter here.

31/10/22 17:22:57 - INFO - [login_flow_backup]: Extracting cookies... 31/10/22 17:22:59 - ERROR - [post_integrity]: Uh-oh, Twitch has detected this miner as a "Bad Bot"

Then the bot works but doensn't seem to claim the +50 bonus points

I'll try with @FarmerBrianW method and report later

[rdavydov]

@realAbitbol that method has nothing to do with bonuses.

Could anyone else please test this new version and report back?

Because it's been working for hours now on my Windows box without any issues.

[PingkeuKeobi]

I am currently testing this, will report back with an edit.

EDIT: I was able to get logged in perfectly fine, however, the miner is detected as a bad bot. I was able to claim the WATCH, currently seeing if I can claim a bonus, will edit when i receive the message.

[realAbitbol]

@realAbitbol that method has nothing to do with bonuses.

Should I ignore the "BAD BOT" error ?

[rdavydov]

Yes, I forgot to mention, the "bad bot" error can be safely ignored, for now Twitch lets us claim bonuses even though they are aware that we are bots. It is purely informational for now.

[PingkeuKeobi]

@rdavydov

Good news! It works! I can claim bonuses and watch points! No errors!

[PingkeuKeobi]

I just received this error in the console, not sure if it means anything bad, but just thought I should include this

31/10 13:59:07 - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000155A8B3BD48>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it')': /session/<mysessionid>

@rdavydov

[rdavydov]

@PingkeuKeobi It's probably nothing if the miner continues mining. Can be ignored.

[NoxRare]

@rdavydov - should I update if I'm on 1.3.4? Does this only fix login? I already have a cookie generated. Should I update or stay?

[rdavydov]

@rdavydov - should I update if I'm on 1.3.4? Does this only fix login? I already have a cookie generated. Should I update or stay?

Only fixes login issues. If you already have a "console" cookie and your miner is claiming bonuses, you can stay on 1.3.4.

[rdavydov]

@XaaRii have you tried the new version? Is everything OK? Please report back, so I can close this issue.

[KraineOpasen]

Version 1.4.0 does not collect +50 bonus points

[rdavydov]

Version 1.4.0 does not collect +50 bonus points

Proof with debug logs or it didn't happen. 😉

[antipatico]

https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/releases/tag/1.4.0

Second approach is working beautifully. Also, I've found what I forgot to do in the first one. Will fix it and test later.

For now, I think we can use the second approach. You will need a display, NO CAN DO on a headless machine!

First of all, install and upgrade requirements: pip install -U -r requirements.txt

* If you don't have cookies, a browser window will open, it will login with your data.

* After that, enter your verification code in the browser and wait for the Twitch website to load, then press Enter in the console.

* Everything should be the same afterwards. It collects points and claims bonuses, hooray! 🙂

Is there a way to login and then port the cookie back to docker?

[XaaRii]

@XaaRii have you tried the new version? Is everything OK? Please report back, so I can close this issue.

Haven't tried it yet, as I have my miner on a headless-only environment. But I guess generating cookies on a different machine and then move it should be sufficient? I don't really like having desktop environment on servers (where it isn't needed most of the time.) That being said, I am not home yet. I'll report back once I'll get to it. ;)

[KraineOpasen]

Version 1.4.0 does not collect +50 bonus points

Proof with debug logs or it didn't happen. 😉

Post a log here? He is 1g

[rdavydov]

@antipatico

Is there a way to login and then port the cookie back to docker?

Yes, there is. Just copy the cookie file! :)

@XaaRii This also answers your question.

[rdavydov]

Post a log here? He is 1g

Try to delete the cookie file, obtain it again with 1.4.0 and see how it goes.