v14: JSONDecodeError when trying to log in

GiladKupfer commented 2 years ago

when I put my detail it gave me this error

"Please log in to continue. Fatal error encountered:

Traceback (most recent call last): File "main.py", line 169, in File "asyncio\base_events.py", line 616, in run_until_complete File "twitch.py", line 423, in run File "twitch.py", line 443, in _run File "twitch.py", line 986, in get_auth File "twitch.py", line 208, in validate File "twitch.py", line 233, in _validate File "twitch.py", line 125, in _login File "aiohttp\client_reqrep.py", line 1119, in json File "json__init__.py", line 357, in loads File "json\decoder.py", line 340, in decode json.decoder.JSONDecodeError: Extra data: line 1 column 128 (char 127)

Exiting...

Application Terminated. Close the window to exit the application."

NaniDaFuck commented 2 years ago

ya same thing

DevilXD commented 2 years ago

Hello o/

It looks like it's Twitch's fault. The passport endpoint started to return 400s with a messed up (duplicated) JSON payload. Here's the raw response that fails to decode:

b'{"error":"Please update your app to continue","error_code":5023,"error_description":"client is not supported for this feature"}{"error":"Please update your app to continue","error_code":5023,"error_description":"client is not supported for this feature"}'

The JSON decoder breaks because there's two root objects instead of one.

Anyway, that's not really the main issue here. Looks like Twitch has started to deal with this client ID as well, and I expect it's only going to get worse from here. It's probably going to get protected by the integrity crap just like the main one did. For now, the logins have stopped working, and there's nothing I can do about it. The official Android app is probably going to use a different endpoint for logins, and thus this client ID is no longer allowed to use the normal endpoint.

NaniDaFuck commented 2 years ago

is there no way to login now :/

Huiaei commented 2 years ago

Perhaps consider manually filling in cookies? I've seen many similar projects use this method.

GiladKupfer commented 2 years ago

Perhaps consider manually filling in cookies? I've seen many similar projects use this method. how do i manually put cookies

Huiaei commented 2 years ago

Perhaps consider manually filling in cookies? I've seen many similar projects use this method. how do i manually put cookies

Sorry, this is just a suggestion. I've opened the cookie.jar and although it has the relevant characters in it, vscode doesn't fully open it, so I don't know where to start with the changes.

DevilXD commented 2 years ago

@Huiaei It's not possible. While I can agree that I've seen some of those "many similar projects" that have used this technique, it can't be applied now. It's because Twitch has made changes that require integrity interaction for the web client ID on desktop, and thus your web browser can't be used - you could scrape the authentication information, but not the integrity token, as it isn't stored in a cookie. Even then, you'd have no real way to refresh the token (which can change at pretty much any time), without having your browser open just for this purpose, and then somehow automate interaction with it, which isn't easily possible either.

The miner now uses an android app client ID to bypass the integrity system, and since there's no way to login as an android app via the PC browser, there's no cookie I could scrape either. It's a different platform, so it's not possible.

EffortlessFury commented 2 years ago

Would this be possible if you were running a headless browser?

DevilXD commented 2 years ago

I'm not sure if you missed my point or not. All headless browsers are currently detected by Twitch as bots and denied any login or drop claiming interaction. The web client was the first to receive it, now the android app has received one as well. Trying to login with anything other than a legit browser or app is not going to work, simple as that. There's no existing workaround for the /protected_login endpoint, and the /login one the miner uses, is only going to produce a token that fails any and every Twitch's integrity check. There is nothing I can do about this, unfortunately. If you haven't seen #40 yet, that almost killed this project entirely, I'd recommend checking it out for more info.

EffortlessFury commented 2 years ago

Ah I see, I didn't realize headless browsers were included in what you were discussing.

LegendAdmin commented 2 years ago

@Huiaei It's not possible. While I can agree that I've seen some of those "many similar projects" that have used this technique, it can't be applied now. It's because Twitch has made changes that require integrity interaction for the web client ID on desktop, and thus your web browser can't be used - you could scrape the authentication information, but not the integrity token, as it isn't stored in a cookie. Even then, you'd have no real way to refresh the token (which can change at pretty much any time), without having your browser open just for this purpose, and then somehow automate interaction with it, which isn't easily possible either.

The miner now uses an android app client ID to bypass the integrity system, and since there's no way to login as an android app via the PC browser, there's no cookie I could scrape either. It's a different platform, so it's not possible.

I'm in the same situation, so there is no way to log in now?

DevilXD commented 2 years ago

@LegendAdmin No, there's not. Using a browser cookie will cause points and drop claiming to error out.

Please, if you're new to the issue, see #40 for more information. This is a continuation of the story, really.

LegendAdmin commented 2 years ago

@LegendAdmin No, there's not. Using a browser cookie will cause points and drop claiming to error out.

Please, if you're new to the issue, see #40 for more information. This is a continuation of the story, really.

I looked, but I think it's different from what I encountered, and my error is exactly the same as this one. However, I solved it now, the process goes like this: I used the old V12 version at the beginning. I used it for a long time, but it kept getting an error some time ago, but I didn't have time to worry about it. Later, I went to Github and looked at it, only to find that 2 big version updates have been made recently. Then I downloaded the latest version, because the old version will report an error, I was afraid that Twitch judged my account as a robot, so I deleted the previous cookies and tried to log in again, but I encountered this error, no matter what No matter how many times I try to log in, the result is the same, finally I restored the previous cookies and it actually works again (so I still don't understand why my account can't log in in the software, but I can use it now)

LegendAdmin commented 2 years ago

@DevilXD I thought I was done, but when I rebooted my computer, there was a new problem. Here is a new error code:

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 423, in run
  File "twitch.py", line 484, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 1059, in gql_request
exceptions.MinerException: GQL error: [{'message': 'failed integrity check'}]

Also, now I can still only log in with the old cookie file, and if I delete the file and log in again, I still only get the error message I mentioned before. I re-read your release description, I think it's possible that the CAPTCHA verification is stuck, I'll try to wait for some time before logging in again to see if it works (perhaps you can consider adding the ability to manually pass the verification in v15, although I don't know if this is possible)

Windows200000 commented 2 years ago

@DevilXD I thought I was done, but when I rebooted my computer, there was a new problem. Here is a new error code:
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 423, in run
  File "twitch.py", line 484, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 1059, in gql_request
exceptions.MinerException: GQL error: [{'message': 'failed integrity check'}]
Also, now I can still only log in with the old cookie file, and if I delete the file and log in again, I still only get the error message I mentioned before. I re-read your release description, I think it's possible that the CAPTCHA verification is stuck, I'll try to wait for some time before logging in again to see if it works (perhaps you can consider adding the ability to manually pass the verification in v15, although I don't know if this is possible)

DevilXD already addressed this. While the cookies work for the login, twitch doesn't allow you to actually mine without the integrity check.

@Huiaei It's not possible. While I can agree that I've seen some of those "many similar projects" that have used this technique, it can't be applied now. It's because Twitch has made changes that require integrity interaction for the web client ID on desktop, and thus your web browser can't be used - you could scrape the authentication information, but not the integrity token, as it isn't stored in a cookie. Even then, you'd have no real way to refresh the token (which can change at pretty much any time), without having your browser open just for this purpose, and then somehow automate interaction with it, which isn't easily possible either.

The miner now uses an android app client ID to bypass the integrity system, and since there's no way to login as an android app via the PC browser, there's no cookie I could scrape either. It's a different platform, so it's not possible.

DevilXD commented 2 years ago

@LegendAdmin If you're getting the "failed integrity check" message, that means you are still using the old cookie and haven't read the important note I've attached under v13 and v14 release notes. And because of this issue, you won't be able to generate a new cookie that gets rid of that error. You're can only blame yourself for not reading properly here, really. I can't help you with it.

LegendAdmin commented 2 years ago

@LegendAdmin If you're getting the "failed integrity check" message, that means you are still using the old cookie and haven't read the important note I've attached under v13 and v14 release notes. And because of this issue, you won't be able to generate a new cookie that gets rid of that error. You're can only blame yourself for not reading properly here, really. I can't help you with it.

The problem was that I deleted the old cookie file at the beginning and then logged back in, however, logging back in reported the error I mentioned at the beginning, so I then tried to use the old cookie file, but instead logged in successfully, and I think all of this, including what happened afterwards, I've described before. I don't know if you have encountered the situation I described, but I think the best way to do it, if it is possible, is to add manual authentication via CAPTCHA, what do you think?

DevilXD commented 2 years ago

I'm kinda lost on what you're saying. It doesn't matter though, because as me and you said said, you've got an error and thus can't use the miner anymore. It doesn't matter what you did, it's dead now and there's no way to bring it back. Deal with it.

Also, please just be patient, I happen to have some real life to take care of as well. If I happen to implement anything useful that helps resolve this issue, I'll let you know.

EffortlessFury commented 2 years ago

I then tried to use the old cookie file, but instead logged in successfully

Logged in successfully without the integrity check error? Because if you're still getting the integrity check error you're not really logging in successfully. But yes, as DevilXD said, you would've needed to login using a newer version, but now logging in with this application is no longer possible whatsoever, so you missed your window to get valid cookies.

DevilXD commented 2 years ago

I'm currently in the process of testing a solution that appears to be working for now, as in, I'm getting an authentication token - hopefully a usable and workable one. The solution involves using https://github.com/ultrafunkamsterdam/undetected-chromedriver and a local Chrome installation, to access the https://twitch.tv/login page, and let the user enter their credentials (and optional 2FA) as they normally would. The script would then intercept the intermediate /protected_login request and replace it's web browser Client ID with the andoird app Client ID, which yields an authentication token that ultimately isn't protected by the integrity system, bypassing the need to deal with it, as the current v14 does already.

At least in theory, in practice, I'd have to code all of that in first. It also introduces an existing Chrome installation as a dependency, which isn't exactly ideal (I'm a Firefox person myself, for example, had to specifically install Chrome just for this), but it's necessary to effectively deal with this issue. Hopefully it won't be too inconvenient for most people.

DevilXD commented 2 years ago

I had a day off and it looked promising, and 6 hours of binging later: https://github.com/DevilXD/TwitchDropsMiner/commit/73a2a6145f83d04e268ead502dd40ce7d76f8b27

I took care of fixing the freeze dependencies, so an executable is buildable and works (at least so far for me). It's too early for a release yet, and I have exactly no idea what will happen if you'd have no Chrome installed (there's no handling for that case either). If you'd happen to get any kind of an error out of this one, or have any other issue while logging in, please just reply below.

Anyone here willing to do so, please test it and tell me how it went for you.

EDIT: Got a first successful drop claim off this method, so there should be no issues. Please give it a try and report regardless.

scryptio commented 2 years ago

@DevilXD Gave it a try for you. Was Working for a bit over an hour

(env) C:\Users\\Desktop\TwitchDropsMiner [master ≡]> python .\main.py

Fatal error encountered:
Traceback (most recent call last):
  File "C:\Users\\Desktop\TwitchDropsMiner\main.py", line 175, in <module>
    loop.run_until_complete(client.run())
  File "C:\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\Users\\Desktop\TwitchDropsMiner\twitch.py", line 351, in run
    await self._run()
  File "C:\Users\\Desktop\TwitchDropsMiner\twitch.py", line 485, in _run
    await asyncio.gather(*(channel.check_online() for channel in acl_channels))
  File "C:\Users\\Desktop\TwitchDropsMiner\channel.py", line 252, in check_online
    self._stream = await self.get_stream()
  File "C:\Users\\Desktop\TwitchDropsMiner\channel.py", line 238, in get_stream
    response: JsonType = await self._twitch.gql_request(
  File "C:\Users\\Desktop\TwitchDropsMiner\twitch.py", line 987, in gql_request
    raise MinerException(f"GQL error: {response_json['errors']}")
exceptions.MinerException: GQL error: [{'message': 'service timeout', 'path': ['user', 'stream']}]

Exiting...
Cannot connect to Twitch, retrying in 1 seconds...

Trying to reconnect in a loop since. Restarting results in the same error. Guess the token got invalidated? Had to remove the cookies.jar and login again.

DevilXD commented 2 years ago

Had to remove the cookies.jar and login again.

Did it start working back up afterwards?

scryptio commented 2 years ago

Yes, after removing and a new login it was working for another 3.5 hours. Came back from a walk, it started watching another game as it seems and now tried to switch to a priority one and is stuck now (but still claiming channel points as it seems). Progress bars in the app are stuck at 0:00:00 and a campaign from another game, too. Weird stuff :-)

EDIT: It says it is mining, but it isn't. Looks like it can't anymore. I will try deleting the cookie again.

DevilXD commented 2 years ago

There's a lot of confusing information here.

It says it is mining, but it isn't.

Are you sure of that though? Twitch has a bug with displaying progress of some drops, where it'll "appear stuck" (no drop progress updates are received) but actually count properly in the background, and then automatically get unstuck at some point later on. You could've ran into that instead. If the watch event is sent, but there's no drop progress response within 10 seconds afterwards it's the watch loop's job to "manually" trigger a minute countdown, so that the timer continues to tick down even when the bug occurs. If no watch events would be sent, the timer would stop counting down and get stuck at the closest whole minute.

but still claiming channel points as it seems

This is also an indication of the watch events still being sent properly, as each event yields +10 points every minute, and +50 claimable points are yielded every ~15 minutes. No watch events being sent would mean no points events being received. Unless you were watching something else on Twitch in the meantime - the miner will actually receive those events as well.

Speaking of watching something on Twitch with the miner in the background - the miner may stop progressing a drop if you're watching something in the browser, on the same account. The reason is that, your browser will also be sending those watch events to Twitch, and those may stop the miner ones from registering completely. It'll assume it ran into the bug and continue ticking minutes down manually, but no actual progress will be made, and once you stop watching in the browser, the first event that'll register will update the drop's progress to the actual one it's supposed to be at. I've personally experienced some funny things happening if you watch the same channel the miner is watching, or at least try to mine the same drop with the browser and miner simultaneously. I remember turning my PC off and going to sleep with 20% progress, only to wake up to 60% progress the next day. Also weird stuff.

tried to switch to a priority one and is stuck now

Always try the Reload option from the Settings tab when this happens, it could be a case where it could miss a "channel going offline" event and it's sending the watch events to an offline channel now (that appears online in the miner). When that happens, no points nor drop progress will be earned, until you either switch the channel manually or it gets to switch on it's own. Or a reload happens of course.

Overall, I don't think it's the cookie problem, seems more like Twitch just struggling to work properly. Or it's your internet connection dropping crucial events for some unknown reason. Either way, please watch out for any traceback being printed out to the output, as having that will help me understand what the problem is and eliminate any bugs, if there would be any. Hope all of this description helps =)

scryptio commented 2 years ago

Are you sure of that though? Twitch has a bug with displaying progress of some drops, where it'll "appear stuck" (no drop progress updates are received) but actually count properly in the background, and then automatically get unstuck at some point later on.

I think we are way over the "twitch has a bug" topic in regards to the percentage. I always check both twitch and the tool. It didn't progress anything for well over 10hours now (I left it running). It seems to collect channel points but that's it.

I have three attempted drops and none of it gained any percentage after the first claimed drop yesterday. And two of the drops probably got percentage due to channel points miner earlier.

I just checked the game it was stuck on visually and opened any stream (cross checked with the user list of the bot) and it is progressing. Restarted the miner and It finished "manually" now, drop immediately claimed. The tool switched the game due to priority right now. Let's see what happens

Speaking of watching something on Twitch with the miner in the background - the miner may stop progressing a drop if you're watching something in the browser, on the same account.

Fair, I didn't know that but this never was an issue before for me. Maybe I was lucky I don't know. But overnight I had only your miner running so it could have progressed.

Overall, I don't think it's the cookie problem, seems more like Twitch just struggling to work properly. Or it's your internet connection dropping crucial events for some unknown reason. Either way, please watch out for any traceback being printed out to the output

Could be true. But on the other hand if the channel points miner is running it is able to "watch" properly. I don't know. My connection definitely isn't dropping anything. The browser and oter tool work as I tested it for 2 hours now. But yours is stuck for whatever reason :-/ Unfortunately no traces left in the logs.

EDIT: As written above the tool claimed my manually watched drop, immediately switched to another game and mined the drop there successfully. After that it switched to another game and is not mining there. It says it is, but it isn't. Whenever I restart, it thinks it is doing progress, until the "Drop:" counts down to 0:00:00 remaining. If you check in the browser, no progress is being made nor did it started trying.

I restarted the application 20min ago from this edit. It picked a game and the countdown began. Cross checking against the browser, nothing happens. I removed the cookie again to retry but it didn't help. I don't know what is going on or how to provide you with what you need to address it.

DevilXD commented 2 years ago

I think we are way over the "twitch has a bug" topic in regards to the percentage. I always check both twitch and the tool.

I think you're not understanding me. It's not something I came up with out of thin air, nor it causes the difference you're suggesting. It's a Twitch bug where the progress just stops updating, both on the Twitch website, and in the miner (because the miner is taking the info from Twitch itself, duh). The progress just stops updating and stays at a set percentage for a while. There's some specific conditions that can trigger it, for example, two similar campaigns changing their state at the exact same time (one expires, the other one becomes active), which I've explained in detail in #15. But I expect those aren't the only conditions it can appear in.

I'm not trying to suggest you're lying or anything, just saying that this is something that exists and has some additional logic in the miner that takes care of it, which is why it can "lie" a little bit and appear as it's still mining, while the Twitch website appears stuck. Once you restart the application, you get the same "broken" progress in both Twitch and the miner.

But on the other hand if the channel points miner is running it is able to "watch" properly.

That's my expectation as well. If you're getting the channel points, that means the watch events are going through properly. If the drops aren't progressing with that, then it's the Twitch just not counting those for some unknown reason. I also had the opposite experience, where not mining anything overnight (after mining the same channel "twice at the same time", once from the miner and once from the browser), gave me like 40% free progress that I didn't really ask nor did anything for. Maybe you could try to stop the mining for a day or two, see if the percentage changes on it's own, and once everything "calms down" sort to speak, try again?

Whenever I restart, it thinks it is doing progress, until the "Drop:" counts down to 0:00:00 remaining. If you check in the browser, no progress is being made nor did it started trying.

This sounds like an actual issue and is something I'd definitely have to look into. I'd definitely need at least the campaign / drop it was trying to mine, and the channel it was trying to do so on. There are some theoretical and tricky edge cases with the current GQL + websocket state keeping, that can miss or overlook some crucial info that'd prevent the mining from commencing, after which the extra logic that handles the bug I explained above, takes over and artificially advances the timer (still sending watch events properly though), despite getting no drop updates from Twitch itself (usually due to the bug).

Again, it'd be best to stop any mining or watching activities for a day or two and let Twitch settle down at a percentage. This "stuck progress" bug usually goes away within ~12 hours max, and on it's own. Never had experienced it longer than that personally, but I'm not rejecting the possibility of it taking days either. From my perspective, the miner just stops receiving drop progression events over the websocket (points still work), even though the watch events are getting sent, and that's it.

DevilXD commented 2 years ago

Also, just to be 100% sure, you're using the current master build, not v14, right?

scryptio commented 2 years ago

I'm not trying to suggest you're lying or anything

Hey, it is fine and I didn't think you suggested that. Just that I did not understood. But I am sure I did. It might be something with how you do it, because I repeat that the channel points miner is able to properly "watch" streams. It never lost the functionality, it just isn't able to claim anymore (or to login... but for whatever reason my cookie is still working, never lost it even after the twitch changes)

Also, just to be 100% sure, you're using the current master build, not v14, right?

Correct. Pulled after your comment, ran the environment setup again and hit build.bat ... But this one isn't working since the modules are missing (needs to be addressed later), so I simply run python main.py

DevilXD commented 2 years ago

this one isn't working since the modules are missing (needs to be addressed later)

If you have any issues with not being able to build the application, please open a separate issue and I'll take a look at it. I've had someone report a missing *.crt file before (FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\...\AppData\Local\Temp\_MEI186202\seleniumwire\ca.crt', but that apparently went away after they've deleted the env folder and recreated it from scratch. If that doesn't help, then please make a separate issue and I'll take a look.

Regarding the issue you're experiencing, I think it's just Twitch doing something weird with your drops progression right now. There is definitely a difference between the points system and the drops system. The points system is able to handle two channels being watched at the same time, with no bigger issues. But for drops, only one drop can be progressed at a time. Switching channels stops the drop progress from being reported for ~5 minutes, but it usually comes back after that, and its progress is advanced by those 5 missing minutes like one would expect. Double watching same channel, or two different channels that'd mine the same drop, appear to not influence the speed of progressing the drop, but as I've mentioned before, I've had some extra percentage appear out of nowhere overnight, after an action like that. Also had the progress get stuck in place after switching campaigns, like described in #15, for good several hours. It'd be no surprise to me if you, by sole accident, triggered some unknown logic on Twitch's side, and now your drops are stuck and not reporting any progress at all.

My suggestion is to just stop watching / mining for a while and let things settle down. Twitch definitely has something running in the background that "archives" these events, otherwise it'd be impossible for me to gain progress overnight (while my PC was off and everything). Maybe they just need a day or two for some system to clear out the events queue or something. I don't know. What I do know, is that, if changing the cookie isn't helping, then there's nothing else one can really do here.

As far as receiving the drop events goes, there is "a" way to tell if it receives a drop update or not. The timeout on the watch event "waiting" for a drop update to come back, is 10 seconds at maximum. If the countdown reaches 0 seconds and then stays there for ~7-8 seconds or more, then you'll most likely see the extra logic handling the timer in action. If it starts counting anew within the first 5 seconds, it usually means a response was received. The problem is that, this can only really be noticed the first time the logic switches from using Twitch websocket events, to using the extra logic - after that, it'll appear to count down perfectly. Once an update is finally received, it does the opposite - skips the last 3-6 seconds of the countdown, when it goes back from extra logic to using websocket events again.

The easiest way to trigger the extra logic to take over, is to change the watching channel. As I've mentioned, the events stop coming back for the next ~5 minutes or so. You should see the timer get stuck at 0 for ~7 seconds shortly after the changing, and then 5 minutes later, have those last 7 seconds skipped right to the next minute.

I've actually went back and dug up a really old screenshot a friend sent me, from like a version v3 or something like that. The GUI was added in version v4, and before that, this miner was a console application. It clearly shows that the updates just stop coming back for 5 minutes after channel switching, but once they do come back, they're with the progress you'd expect: obraz

And finally, if you'd want to get some actual logs to go through, you could try adding -vv --log --debug-ws as the run arguments of the miner. This should cause it to create a --log file next to the executable, with INFO level information logging (-vv) and extra websocket debug info (--debug-ws), that should hopefully include all drop update events as they're received. I could probably spend some time and update the logging backend of the miner to better accommodate debugging situations like this. I'll try doing so and will let you know if/when that happens, so you can pull a new version for testing. Before posting the log, please ensure there's no authentication token in the log (there's a high chance it'll be there), otherwise you may have some unexpected visitors doing stuff to your account. If you'd want to pass the log securely to me, I'm available on Discord under DevilXD#4437, and accept friend requests from people I don't share any servers with. Not sure if github has a direct message option, but it'd also work for it too.

hadenting commented 1 year ago

Have you deviated from the topic? I want to know how to log in successfully when I encounter JSONDecodeError！Could it be that I used a proxy? But I can connect to twitter.

DevilXD commented 1 year ago

@hadenting Not really, the recent messages are about having issues with the latest master build, which fixes this exact issue. You'd need to run the application from source though, as there's no official nor unofficial executable release for v15 yet. If you'd be interested to do so, see: https://github.com/DevilXD/TwitchDropsMiner#advanced-usage--build-instructions

Albertyangbu commented 1 year ago

@DevilXD After open the chrome box, it can't connection to the twitch login page, and show me this

502 Bad Gateway
ProtocolException('Server connection to (\'twitch.tv\', 443) failed: Error connecting to "twitch.tv": [WinError 10060] ç”±äºŽè¿žæŽ¥æ–¹åœ¨ä¸€æ®µæ—¶é—´åŽæ²¡æœ‰æ£ç¡®ç”å¤æˆ–è¿žæŽ¥çš„ä¸»æœºæ²¡æœ‰ååº”ï¼Œè¿žæŽ¥å°è¯•å¤±è´¥ã€‚')

Maybe because I used a proxy? but I can login to twitch with chrome used alone.

Luckz commented 1 year ago

'm not sure if you missed my point or not. All headless browsers are currently detected by Twitch as bots and denied any login or drop claiming interaction.

Well yes. Did you research if this perhaps has something to do with the stuff Twitch stores in local storage? (examples: local_storage_device_id, local_copy_unique_id, twilight.sessionID, local_storage_app_session_id, sentry_device_id)

Over in the headless browser world, it works rather tolerably if I first make a user profile with a 'real' Chrome and then continue running it headlessly. Once I get an integrity failure (f.ex. because of trying to claim 10 drops at once, automatedly), I of course need a new fresh login.

While the cookies work for the login, twitch doesn't allow you to actually mine without the integrity check.

Well no. The core part of mining (IMO) is making drop progress, and that can be done even if you fail integrity checks. As DevilXD wrote, it's login and drop claiming that can not. Hence some people use a Twitch miner to gain drop progress, and a separate human-use browser that claims the drops (usually via browser extension).

DevilXD commented 1 year ago

@Luckz You're like, super late to the discussion by now. That comment of mine was based on the information from #40, specifically:

https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1261361605 https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1261908399

Regardless, this issue has been almost fixed by now per what I described in https://github.com/DevilXD/TwitchDropsMiner/issues/68#issuecomment-1298434686, and it stays open solely because there's no official release covering it yet. There's still some extra work to be done here, like taking feedback on the login process, fixing bugs and building errors, etc.

Also, yes, mining itself isn't impaired by the integrity system, only login (now bypassed via Undetected Chrome Driver) and drop claiming (also bypassed via android app client ID) are.

DevilXD commented 1 year ago

@LiquidMX Any of the v13.dev versions is highly discouraged from usage, as they use the web client ID and none of the integrity handlers there can make it past the integrity system. If you're looking for a version that works, the current master branch (V15.dev) has a redesigned login system via Chrome and an undetectable webdriver. If you're interested in trying it out, you'd have to build the executable yourself (or just run it from source), by following the https://github.com/DevilXD/TwitchDropsMiner#advanced-usage--build-instructions I can also assist you with any step of this via DevilXD#4437 on Discord, or if you'd simply open a separate issue and ask for a more in-depth explanation there.

Speaking of v15.dev, I'm currently in the middle of testing this new system, and just making sure that all edge cases are handled and it's not going to break once I release it. It should be ready in a week or two (or more), depending on my free time availability.

hadenting commented 1 year ago

I have set a proxy for the software, but it seems that the proxy is not used for chrome login, so I cannot log in through the proxy. Can you add the software proxy to chrome login Not secure https://twitch.tv/login 502 Bad Gateway TlsProtocolException('Cannot establish TLS with twitch.tv:443 (sni: twitch.tv): TlsException("SSL handshake error: SysCallError(10054, \'WSAECONNRESET\')")')

DevilXD commented 1 year ago

@hadenting Can't you just open the Chrome's settings and add the proxy in manually? The browser window that's open isn't much different from a normal Chrome window, minus the fact that I can control it and look up stuff like current URL and cookies from the application, so the procedure should be the exact same as with normal Chrome. It should work for now.

That being said, I will try researching if it'd be possible to start the Chrome instance with Proxy settings already in place, so one doesn't have to do that every time.

Huiaei commented 1 year ago

I have built cc4cbc5179de2c905b14ed2b37456dc3f9e22007 using github action. i noticed that chrome is used, but instead of installing chrome in my computer, i am taking the portable version. Is it considered possible to customize the chrome.exe path?

Here is the log.

Opening Chrome...
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 179, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 351, in run
  File "twitch.py", line 370, in _run
  File "twitch.py", line 913, in get_auth
  File "twitch.py", line 136, in validate
  File "twitch.py", line 161, in _validate
  File "twitch.py", line 107, in _login
  File "gui.py", line 506, in ask_login
  File "gui.py", line 1970, in coro_unless_closed
  File "concurrent\futures\thread.py", line 57, in run
  File "gui.py", line 497, in <lambda>
  File "seleniumwire\undetected_chromedriver\webdriver.py", line 32, in __init__
  File "seleniumwire\webdriver.py", line 41, in _setup_backend
  File "seleniumwire\backend.py", line 24, in create
  File "seleniumwire\server.py", line 28, in __init__
  File "seleniumwire\utils.py", line 159, in extract_cert_and_key
  File "pkgutil.py", line 637, in get_data
  File "PyInstaller\loader\pyimod02_importers.py", line 344, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Hu\\AppData\\Local\\Temp\\_MEI63562\\seleniumwire\\ca.crt'

hadenting commented 1 year ago

@DevilXD My chrome can use my proxy, but the chrome opened by the software cannot be used. I think other people will also encounter this problem. I checked the chrome driver and can set the proxy directly through code. It has taken me a long time to use this software as happily as before. Maybe you can add a function of manually entering browser tokens to log in easily

DevilXD commented 1 year ago

@Huiaei That error is already on my TODO list that you can find in #73. The problem is that it appears only on PCs that aren't involved in development (didn't run selenium from source), which I missed because I have to test the code by running the source before building, which places that file at that path and masks the error.

@hadenting You're having trouble using Chrome as a browser? I don't get it, it goes the same way as you'd normally login into Twitch. There is some waiting involved, especially at the end, but there's no easy way to reduce this wait time, as it doesn't depend on my (the Selenium webdriver dictates when it finishes loading a page). I added a way to speed up the process by clicking on the Login button again, but that's all I can really do.

Regarding pre-set proxy support, it's also on my TODO list at #73 already, you'll just have to wait for it.

Also no, there will never be an option to enter a "browser token", simply because of these two reasons:

I find the solution of having to dig up browser cookies by yourself and copying some cryptic sets of letters and numbers very unfriendly to the user. Even if you've did it before in other software, I'm still very much against anyone having to do that - if I were to use any form of this, it'd be a fully automated no-touch solution instead.
Regardless of the above, the current browser client is protected by the integrity system, and thus cannot be used at all, even if there would be a possibility to enter it. You can manually edit the cookies.jar file and replace the token with your web one to see for yourself (it's already possible to enter the token via this method), but you won't be able to get the miner past the first drop claim, as the integrity system will reject the request and cause the "integrity check failed" error in the application. The current solution the miner uses is to login as the android app, which isn't protected. There's no way to get this from a PC browser without intrusive request manipulation (that you can see being utilized at the current gui.py#L448-L459).

Huiaei commented 1 year ago

Okay, I see it. I'll wait. Thank you for your work.

pree commented 1 year ago

I'm also unable to log in with the chrome-browser with the current master branch:

grafik

I've tried using another bot before, so maybe my account could be flagged, though.

DevilXD commented 1 year ago

@pree This is known, I've reported the issue to the undetected-chromedriver team: https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/897

Windows200000 commented 1 year ago

Why does this have the "Solved for the next release" label, when the undetected chrome driver doesn't pass the twitch integrity system (as @pree pointed out and I experienced myself with current master)?

DevilXD commented 1 year ago

Well, the general gist of it was that it was solved for the next release, but someone else threw a wrench into the gears, and now it's kinda "pending" instead. However, I'm working on an additional something that might turn this back into being solved, even without the chromedriver - turns out Twitch pulled off the same thing as they did last time, and the standard login endpoint may work like how it used to again, without what the 3rd comment in this issue says (5023: client is not supported for this feature). The same thing happened previously as well, although didn't get us anywhere since the generated cookie was protected by the integrity system later down the line. This one still isn't, and it's just a matter of bringing back the old login flow, with Chrome login flow as a backup, and it should be okay.

The only problem is little available time I have every day, with all the usual life things happening around me. There's the weekend coming, where I'll hopefully be able to push something useful out for testing. Please have patience and stay tuned.

Windows200000 commented 1 year ago

Don't feel pressured to hurry. You are doing this for free, which I appreciate, but you have no obligation to even keep this going, so if other stuff gets in the way don't feel too stressed about it... <3

matarife123 commented 1 year ago

i also got this error, after closing and starting several times the program, magically it worked again

JourneyOver commented 1 year ago

i also got this error, after closing and starting several times the program, magically it worked again

yea haha, the past few days I've had troubles getting the application to login.. just tried again a minute ago and magically it logged in just fine and is now back to working order from what I can tell (will have to wait a while and see if anything else happens).

edit: Should mention I'm still running v14 since I can't get the master branch to work for me due to me not having normal chrome (I'm using a browser called catsxp which is chrome based and it's portable) and I don't really feel like downloading normal chrome for just one application..

edit2: looks like it worked up until the point that it grabbed a drop, and then it threw an error:

Claimed drop: Pearl SMG (3/4)
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 423, in run
  File "twitch.py", line 470, in _run
  File "twitch.py", line 1084, in fetch_inventory
  File "twitch.py", line 1059, in gql_request
exceptions.MinerException: GQL error: [{'message': 'service timeout', 'path': ['currentUser', 'inventory', 'dropCampaignsInProgress']}]

Exiting...

Application Terminated.
Close the window to exit the application.

closing the application and opening it back up I'm still able to login and it continues to work, will keep an eye on things.

DevilXD commented 1 year ago

Hmm, well... Yes, this is actually possible, since as I've mentioned, it looks like Twitch has gone back to using the normal login procedure (that v14 uses), and I'm currently working on reimplementing it in v15, together with Chrome login and additional login that'll try the standard login first, with Chrome as a fallback in case Captcha or any other problem happens.

@JourneyOver That looks like a simple GQL timeout due to Twitch having a temporary hiccup, I can add a retry to it when that'd happen again in v15. Cheers for the traceback ^^

DevilXD / TwitchDropsMiner

v14: JSONDecodeError when trying to log in #68