Application doesn't work anymore

ThisIsCyreX commented 1 year ago

Hello!

Miner works fine until the drop is done and gets to the claiming phase:

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 646, in run_until_complete
  File "twitch.py", line 170, in run
  File "twitch.py", line 228, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 130, in _claim
KeyError: 'data'

Exiting...
2022-09-14 00:32:58.981:       INFO:    Websocket[0] stopped.

Application Terminated.
Close the window to exit the application.

I noticed it with this drop: and this: (I claimed them manually)

Both Ubisoft titles. Not sure if this is important.

Nothing in the log except Response 200, keep-alive

Thank you :)

DevilXD commented 1 year ago

It appears that Twitch has made a change to their web client: obraz

The claiming operation now requires a bunch of new headers: obraz

There's a new GQL endpoint associated to obtaining the Client-Integrity value, which I'm assuming is a part of the application state now (just like the authorization token): https://gql.twitch.tv/integrity It probably comes with it's own set of headers that all need to be send in order to pass the integrity check, like the X-Device-ID: cookies.unique_id header I just found in one of the JS files.

Unfortunately, all of this means that there needs to be a bunch of more additional logic designed and implemented, for which I have to find the time for. There is also a possibility that during the design process, I'll discover a captcha-like mechanism that could kill this project - let's hope it's just a bunch of new hops one needs to go through and that's it.

For now, I recommend mining the drops the usual way. I'll look into this as soon as I'll be able to.

EDIT: More information about the /integrity endpoint: https://github.com/mauricew/twitch-graphql-api#integrity

DevilXD commented 1 year ago

@ThisIsCyreX Also, if you'd be interested, regarding this:

Nothing in the log except Response 200, keep-alive

Try it with a --debug-gql argument to see the GQL errors returned as responses.

josephisticated commented 1 year ago

is it possible to disable auto claiming so the program won't crash?

DevilXD commented 1 year ago

@josephisticated The problem is, that's not the only thing that's broken - every GQL operation is. One of the most basic and important GQL operations is fetching the current drops inventory, which powers the entire channel fetching, selection and watching logic. Without that, you won't get the channel list populated, and thus nothing will work.

I'm working on a solution which I'm 80-90% into finishing by now, tired to binge it yesterday. Here's everything I've done so far: https://github.com/DevilXD/TwitchDropsMiner/commit/01729761d8efb3d6824bec7f334fa5ad86dc9b04

DevilXD commented 1 year ago

I have a semi-working version that you should be able to use for the time being - please do note that I want to release a proper v13 as soon as I will be able to iron out some bugs with the current solution, as it still seems to end up with this error sometimes. Currently waiting for a couple of drop-claim cases, with GQL debug and log enabled, to see what exactly is returned that causes the error. If you would encounter it while testing too, please leave a comment with the traceback and information if you had any successful claims on that particular application run, or did it break on the first claim it tried.

Download moved: https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1250123813

EDIT: Oh, and it's possible you can get a KeyError: unique_id of some sorts, which will happen if your cookie file is missing one. Had this happen to me during testing. If so, just delete the cookie file and login again - the login flow has been rearranged a little so that it'll fill up that value in the cookie file before it's used, which should prevent this error from happening. A note about this behavior should find it's way into the eventual v13 release too.

Diefishfish commented 1 year ago

still error :-(

Fatal error encountered:

Traceback (most recent call last): File "main.py", line 169, in File "asyncio\base_events.py", line 646, in run_until_complete File "twitch.py", line 380, in run File "twitch.py", line 437, in _run File "inventory.py", line 191, in claim File "inventory.py", line 109, in claim File "inventory.py", line 125, in _claim File "twitch.py", line 996, in gql_request exceptions.MinerException: GQL error: [{'message': 'failed integrity check'}]

DevilXD commented 1 year ago

Yes, this is related to the integrity system Twitch has added recently, and the exact thing I'm trying to still fix. But hey, at least you've got a different error than me. This is the actual error I should be getting, yet I'm still getting the one OP reported in the first message here (KeyError: 'data'), unless Twitch changed something on their side and it finally works like it should.

Will keep testing, thank you for letting me know :slightly_smiling_face:

DevilXD commented 1 year ago

It turns out that I've made a huge mistake, and the integrity verification is performed ~70 times (total amount of all campaigns) at application startup, which is also probably what's causing the bug. Please upgrade your v13.dev version to this one ASAP, before Twitch gets real angry at us :no_mouth:

Download moved: https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1250123813

Diefishfish commented 1 year ago

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 382, in run
  File "twitch.py", line 439, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 998, in gql_request
exceptions.MinerException: GQL error: [{'message': 'failed integrity check'}]

DevilXD commented 1 year ago

@Diefishfish Please, only the traceback is more than enough. Is this from the newest v13.dev version I posted today, or from the previous one?

MadLongTom commented 1 year ago

hi,I'm using the latest one u post 1h ago and got no error,but the inventory frame is empty after all getting stuff

MadLongTom commented 1 year ago

also,the channel list is empty.

MadLongTom commented 1 year ago

i got the gql debug output but found no error there,i can upload it to u if u need it.

MadLongTom commented 1 year ago

@DevilXD

DevilXD commented 1 year ago

@MadLongTom If you've got no error, that means it's working properly. If the inventory tab is empty, try playing around with the filter, by checking "Expired", "Excluded" and "Finished". It being empty usually means you'll have no channels either - there's just nothing to mine for you for now.

matarife123 commented 1 year ago

https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1250030009

this version works for me

MadLongTom commented 1 year ago

@DevilXD my fault,my test account didnt link to any platform so that no any stream i can join to get drops

DevilXD commented 1 year ago

v13.dev version 3. I started numbering those to make sure I'll know which one you're using when you report back, so this one is called v13.dev.3.

What I expect might be happening, is that - since this new integrity token has only 5-6 hours of lifetime, after which it becomes invalid, it could be that it can expire right before a request using it is made. This is even more significant when we take the infinite retry loop each request can end up in into account. Since the request logic is quite unaware of the context with which the request is being made (and doesn't normally know that there's no point retrying a request that uses expired information), I've attempted to change said logic to allow for the infinite loop interruption in case such expiration happens.

Because this condition can really happen anywhere in the application, a bunch of changes have been made so that the entire application will restart it's main loop and state internally - this is just like a normal reload, but also includes things like entirely restarting the websocket pool and clearing the authentication state, which means it'll need to reauth the app back from the cookie file. Since this can happen only as a corner case result, and doesn't do much more than a normal reload, it should be fine to happen from time to time.

This should prevent the integrity failures like in https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1250036996 for good, but still - if you'd encounter an issue while using this version too, please reply here with the traceback, name of the campaign/drop you were mining at the time, and approximate runtime duration of the application (X hours format is fine, even if approximate).

Download moved: https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1252826458

Last commit: https://github.com/DevilXD/TwitchDropsMiner/commit/1810d802b4eed2984a2eaebcccef7bd27b5289bf

MadLongTom commented 1 year ago

@DevilXD Now it works fine, except for the occasional intergrity check error output, but it doesn't affect usage😄

DevilXD commented 1 year ago

except for the occasional intergrity check error output

You're not supposed to be getting these. Mind posting the traceback here?

Also, how it doesn't affect usage? Doesn't the application stop when it happens?

MadLongTom commented 1 year ago

it happened both when watching and claiming,it wont stop when watching,but stopped when claiming.i modified inventory.py to let the response of claim state always be True,so i only need to claim it manually or use another webdriver script.

MadLongTom commented 1 year ago

except for the occasional intergrity check error output

You're not supposed to be getting these. Mind posting the traceback here?

Also, how it doesn't affect usage? Doesn't the application stop when it happens?

but i got a multithread error when using --no-run-check cmdline:

Traceback (most recent call last):
  File "main.py", line 184, in <module>
  File "asyncio\base_events.py", line 642, in run_until_complete
  File "twitch.py", line 327, in shutdown
TypeError: stop() got an unexpected keyword argument 'clear_topics'

it also occured in singlethread when closing GUI.

DevilXD commented 1 year ago

but i got a multithread error when using --no-run-check cmdline:

This is kinda impossible as the current v13.dev.3 I've posted higher up, cannot produce this error. The method does have that particular keyword argument already defined properly there: obraz

If you're using a different version that v13.dev.3, then please update to the most recent one. If you're using a v13.dev.3 that's been compiled by another person and they've messed something up in their files (also a possibility that I know might be the case here), then please contact them instead of me. If you're using multiple copies of the miner, then please don't forget to update all copies.

MadLongTom commented 1 year ago

I compiled it manually by deleting build folder first,then run build.bat.All files are latest and code in line 327 is same.

DevilXD commented 1 year ago

@MadLongTom You then need to ensure that the stop function of the websocket doesn't differ either: https://github.com/DevilXD/TwitchDropsMiner/blob/master/websocket.py#L343

If you're using git, just pull all changes or clone again the repository, and it should be good.

DevilXD commented 1 year ago

v13.dev.4 - I've studied Twitch's core JS code and found this snippet for calculating integrity token expiration:

(i = Math.round(.9 * (n.expiration - Date.now()))) < || (this.logger.debug("Refreshing in ".concat(Math.round(i / 1e3 / 60), " minutes"))

Now, I'm not a JS guy, but it's also a programming language and most languages are similar, so as far as I understand it, the Twitch site shortens the expiration timestamp by about 10% of the returned duration. If this is done server-side as well, any request done within those 10% could return an error like everyone's experiencing. Twitch essentially lies to itself, and then corrects the lie in the code - kinda strange but so be it.

This 4th dev version has some code to follow this exact same logic of shortening the timestamp, as well as having some extended exception information that should be printed out, when even this won't help the cause here. If you'd still encounter the issue, please reply with a traceback here.

Download moved: https://github.com/DevilXD/TwitchDropsMiner/issues/40#issuecomment-1252826458

Last commit: https://github.com/DevilXD/TwitchDropsMiner/commit/921b3b91703940edaea83b95f6aa85551b164af0

ThisIsCyreX commented 1 year ago

I just saw I got an error with v13.dev.4

[...]
No available channels to watch. Waiting for an ONLINE channel...
Earned points for watching:  10, total: 10
2022-09-20 05:57:15.416:      ERROR:    Exception in task
Traceback (most recent call last):
  File "utils.py", line 79, in wrapper
  File "twitch.py", line 978, in process_points
  File "twitch.py", line 1195, in claim_points
  File "twitch.py", line 1061, in gql_request
exceptions.MinerException: GQL integrity error: now=2022-09-20 03:57:15.416353, request=2022-09-20 03:13:16.397231, response=2022-09-20 03:13:16.687202, expires=2022-09-20 10:25:16.711320
Earned points for watching:  10, total: 40
Claimed drop: Ouroboros Crown Crate (1/1)
No available channels to watch. Waiting for an ONLINE channel...

But it works, it stil idles without crash :) The app now ran easily 24h

Thanks 👍

DevilXD commented 1 year ago

There's something funky going on. It can be either Twitch being unhappy, or yet another bug (which I doubt cos there's already so many checks in place), but it can also be something silly, like I describe below.

request =  2022-09-20 03:13:16.397231
response = 2022-09-20 03:13:16.687202
now =      2022-09-20 03:57:15.416353
expires =  2022-09-20 10:25:16.711320

request and response are when the integrity token got requested, received and stored. now was the time at which the error happened. expires is the time the stored token was supposed to expire at, yet it's already invalid.

No available channels to watch. Waiting for an ONLINE channel...
Traceback (most recent call last): ...

This reminds me of a possibility of a system I saw at another API, where a special session token would expire prematurely, if there was no request using it in the past 15 minutes. Essentially, you could start a session, get a token, and then keep using it for literal days, but as soon as you stopped using it for more than 15 minutes, it'd expire and any request trying to use it from then on would error out. The solution was to just start another session when this happens, of course.

There actually might be a system like this here. When the application is waiting for something to happen, it doesn't do any GQL requests until the maintenance task reloads the inventory (up to once per hour). This would explain many people getting the error on their first claim, if they started the application way beforehand, where the expiration time wasn't over, but the token wasn't used so it just expired shortly after, and the next claim trying to use it caused an error. It's weird though, because the maintenance task also ensures no unclaimed points and does a GQL request to do so, which would still use the token every ~15-30 minutes at most. It seems like that could already be too long and the token may expire anyway.

I have an ultimate way of dealing with this though, already setup in the application - the GQL request can just obtain a new token once the current one is detected to be expired. It's also already in the code, just needs to be uncommented: https://github.com/DevilXD/TwitchDropsMiner/blob/f93da9349393b37e4836313d1657d2ab48c9cc53/twitch.py#L1052-L1068 I changed the logic around so that the entire application won't get reloaded for a silly reason like this. This should deal with the issue for good. I'm making this a thing for the upcoming v13.dev.5 to test.

DevilXD commented 1 year ago

v13.dev.5 - done like I said so. The error is now turned into a non-fatal output message, which is also easy to miss - but I'm hopeful you're still gonna help me by providing whatever it outputs from time to time. It'd be really cool to have a couple instances of it, before proper v13 release.

Download has been removed, because this version doesn't work anymore. See the releases page for the latest official release, or check out the comments below for any additional dev version.

Last commit: https://github.com/DevilXD/TwitchDropsMiner/commit/53d232daa98177313fd8f6f2c9eead58207e89bf

ThisIsCyreX commented 1 year ago

Got one! 👍 Ran almost 24 hours.

[...]
No available channels to watch. Waiting for an ONLINE channel...
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 418, in run
  File "twitch.py", line 465, in _run
  File "twitch.py", line 1093, in fetch_inventory
  File "twitch.py", line 1039, in gql_request
  File "twitch.py", line 981, in get_auth
  File "twitch.py", line 198, in validate
  File "twitch.py", line 279, in _validate
exceptions.MinerException: Twitch considers this miner as a "Bad Bot". Try deleting the cookie file and try again.

Exiting...
2022-09-21 22:32:44.794:       INFO:    Websocket[0] stopped.

Application Terminated.
Close the window to exit the application.

Deleting the cookies.jar and a fresh login will show the samce traceback.

Not working anymore for me 🤔

DevilXD commented 1 year ago

I knew this was probably gonna happen eventually. The integrity token is actually some base64-encoded JSON with some trash at the end, which since I knew about it, I added some double-checking code as a precausion.

Code: https://github.com/DevilXD/TwitchDropsMiner/blob/53d232daa98177313fd8f6f2c9eead58207e89bf/twitch.py#L269-L282

The decoded token:

{
    "client_id":"kimne78kx3ncx6brgo4mv6wki5h1ko",
    "client_ip":"...",
    "device_id":"...",
    "exp":"2022-09-22T13:03:01Z",
    "iat":"2022-09-22T05:03:01Z",
    "is_bad_bot":"false",
    "iss":"Twitch Client Integrity",
    "nbf":"2022-09-22T05:03:01Z",
    "user_id":"94275183"
}

As you can see, there's this "is_bad_bot":"false" flag there, which ends up being "true" when the miner requests an integrity token (this one is from a browser), which means that Twitch actively knows that it's an automated script accessing the API, not a browser. The point is that, to decode this information, an integrity token has to be returned - so it's technically possible to still make it mine using this token, you'll just be actively letting Twitch know that there's a bot accessing your account, which may end up badly. The current logic throws an exception if this flag is detected (per the code above), to prevent you from mining with it as an account safety precaution.

This "grounds" this project yet again, until I can figure out how to make it "false". If you're brave enough, you can find the above code, and comment it out to disable the check, but you're doing so at your own risk. I'm not even sure if Twitch won't reject GQL requests when that flag is set, and I don't really want to try and find that out myself.

DevilXD commented 1 year ago

Apparently it could be because the Client-Version header changes very frequently. Not sure how frequent it is, but right now, it's only fetched once and stored for the entire application run duration. If it's more often than 24 hours (which it probably is), I'll need to add a way to refresh this value every so often.

I'll try to look at this later today.

luciferseraph commented 1 year ago

Per this doc, it looks like it's a new feature deployed in this month.

jldhrj commented 1 year ago

13.DEV.5 atal error encountered:

Traceback (most recent call last): File "main.py", line 169, in File "asyncio\base_events.py", line 616, in run_until_complete File "twitch.py", line 418, in run File "twitch.py", line 438, in _run File "twitch.py", line 981, in get_auth File "twitch.py", line 198, in validate File "twitch.py", line 226, in _validate File "twitch.py", line 170, in _login exceptions.LoginException: Oops! We encountered an unexpected error. Please try again.

DevilXD commented 1 year ago

Yes, it looks like Twitch has put a lock not only on their GQL endpoint, but on the passport one as well. The current login endpoint the application uses, doesn't exist anymore, so it's going to keep reporting errors like this.

The amount of work that needs to be done here is quite a bit, and I'm talking about the implementation only. Knowing what to implement, and if it'd even be possible to implement (can't solve captchas) is it's own problem that will take literal weeks to determine. For now, unfortunately, I recommend going back to "mining" in the old fashioned way.

DevilXD commented 1 year ago

I've managed to track down this: https://k.twitchcdn.net/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/p.js

Not sure how long is this URL going to remain valid, but it looks to me like some kind of obfuscated, self-modifying JavaScript code. Here's some non-obfuscated keywords from it:

'workTime'
_unhandledRejectionFn
encodedClientToken
"fantasy-land/empty"
FrontendErrorCode.JsSdkInterceptXhrError
interceptForms
FrontendErrorCode
defineProperty
prototype
formHasTag
tagForm
createHiddenInputElement(..., "submit")
addEventListener
getSolvedChallengeIfNeeded
getSolvedChallengeIfNeededSync
requestChallengeSync
getInterceptDataSync
interceptConfig
JS_SHA256_NO_NODE_JS
JS_SHA256_NO_COMMON_JS
requestMatchesEndpointDefinition
iframeManager
cryptoChallengeEnabled
checkAllDomainsReady
document.dispatchEvent
managedIFrames
addServerOffset
parseSDKMessage
platformInputs
decorateFormSyncIfPossible
getActionAndMethodFromForm
interceptSubmitFunctionSync
addInterceptDataToForm
FIELD_NAME_CHALLENGE_DATA
"@@transducer/init"
"__esModule"

These mentions of "challenge" point out to me that this is probably the new "entry point" when it comes to both GQL and passport authentication. The result is a new header that needs to be added when requesting an integrity token:

x-kpsdk-cd: {"workTime":1663952254845,"id":"...","answers":[5,4],"duration":11,"d":-4692,"st":1663962234862,"rst":1663962230179}

Trying to request an integrity token without this header, or even something as simple as re-sending the same request a second time, causes the returned token to contain the is_bad_bot flag set to "true", which can't be used for mining.

Since I'm not a JS programmer, going anywhere with this is quite a lost cause. Simpler obfuscated code can usually be bypassed by just running it and recording the output, but this one seems to use iFrames and hidden forms, which require a typical browser environment. De-obfuscation is out of question since it'd take a really long time and everything would be rendered worthless the moment Twitch would decide to use a different p.js file, which the URL appears to heavily suggest.

I'll need to think heavily about what to do in this situation, but it may mean the end of this project. There's nothing much any of us can do, besides waiting and playing this game by Twitch's rules for now.

hrmlsr commented 1 year ago

V12 worked as well,but 13.DEV.5 atal error encountered QQ截图20220924090451

xmhyysd commented 1 year ago

v12 no

hrmlsr commented 1 year ago

after get one drop,V12 also down

DevilXD commented 1 year ago

@hrmlsr Your v12 is going to stop working soon, since the validation endpoint keeps clearing everyone's cookies, and the login endpoint no longer exists. Without the login endpoint, there's no way for me to create or recreate the cookie file, and without that, you can't authenticate with Twitch or use the miner. Surprisingly, the mere "watching" a streamer action remains untouched, since that interacts with Spade and not GQL, but claiming bonus points or claiming drops does, and is thus no longer possible.

This isn't some kind of an "issue" with v13, it's just how Twitch decided to change things. v12 does not have any of the additional checks v13 does, so it's going to try claiming bonus points or claiming a drop, fail miserably, and then Twitch will invalidate your authentication token, leading to you getting the same error as the guy above next time you start the miner.

I know where the issue is, but now I need time. Lots of it. There's at least two ways this project can go forward with, but it's going to be a heavy decision to make, and thus I want to try some things before I do so. For now, you'll have to go back to mining the usual way. I'm sorry but I can't change this.

scryptio commented 1 year ago

v13.dev.4 commit 921b3b91703940edaea83b95f6aa85551b164af0

Earned points for watching:  10, total: 51410
2022-09-23 03:52:13.317:      ERROR:    Exception in task
Traceback (most recent call last):
  File "utils.py", line 79, in wrapper
  File "twitch.py", line 978, in process_points
  File "twitch.py", line 1195, in claim_points
  File "twitch.py", line 1061, in gql_request
exceptions.MinerException: GQL integrity error: now=2022-09-23 01:52:13.317426, request=2022-09-22 23:03:01.728101, response=2022-09-22 23:03:01.979809, expires=2022-09-23 06:15:01.184381
Earned points for watching:  10, total: 51420
Earned points for watching:  10, total: 51430
PaladinsGame goes OFFLINE, switching...
Earned points for watching:  10, total: 85790
Earned points for watching:  10, total: 85800
Earned points for watching:  10, total: 85810
2022-09-23 04:16:08.131:      ERROR:    Exception in task
Traceback (most recent call last):
  File "utils.py", line 79, in wrapper
  File "twitch.py", line 978, in process_points
  File "twitch.py", line 1195, in claim_points
  File "twitch.py", line 1061, in gql_request
exceptions.MinerException: GQL integrity error: now=2022-09-23 02:16:08.130946, request=2022-09-22 23:03:01.728101, response=2022-09-22 23:03:01.979809, expires=2022-09-23 06:15:01.184381
Earned points for watching:  10, total: 85820
Earned points for watching:  10, total: 85830
Earned points for watching:  10, total: 85840
Earned points for watching:  10, total: 85850
Earned points for watching:  10, total: 85860
Earned points for watching:  10, total: 85870
Earned points for watching:  10, total: 85880
2022-09-23 04:55:15.727:      ERROR:    Exception in task
Traceback (most recent call last):
  File "utils.py", line 79, in wrapper
  File "twitch.py", line 874, in process_drops
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 1061, in gql_request
exceptions.MinerException: GQL integrity error: now=2022-09-23 02:55:15.727128, request=2022-09-22 23:03:01.728101, response=2022-09-22 23:03:01.979809, expires=2022-09-23 06:15:01.184381
Earned points for watching:  10, total: 85890
Earned points for watching:  10, total: 85900
Earned points for watching:  10, total: 85910
Earned points for watching:  10, total: 85920
Earned points for watching:  10, total: 85930
Earned points for watching:  10, total: 85940
Earned points for watching:  10, total: 85950
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 642, in run_until_complete
  File "twitch.py", line 419, in run
  File "twitch.py", line 480, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 1061, in gql_request
exceptions.MinerException: GQL integrity error: now=2022-09-23 03:30:21.236564, request=2022-09-22 23:03:01.728101, response=2022-09-22 23:03:01.979809, expires=2022-09-23 06:15:01.184381

Exiting...

Application Terminated.
Close the window to exit the application.

EDIT: Bad bot warning now. Cannot login again exceptions.LoginException: Oops! We encountered an unexpected error. Please try again. and will try again after ip change in a few hours...

DevilXD commented 1 year ago

@scryptio In case it wasn't entirely clear from what I said above - this application is dead. Continuing to use it will lead to errors that I cannot fix, so posting them here is literally pointless.

You're only risking Twitch banning your account at this point. Do not use any version of the miner.

PockySweet commented 1 year ago

@hrmlsr Your v12 is going to stop working soon, since the validation endpoint keeps clearing everyone's cookies, and the login endpoint no longer exists. Without the login endpoint, there's no way for me to create or recreate the cookie file, and without that, you can't authenticate with Twitch or use the miner. Surprisingly, the mere "watching" a streamer action remains untouched, since that interacts with Spade and not GQL, but claiming bonus points or claiming drops does, and is thus no longer possible.

This isn't some kind of an "issue" with v13, it's just how Twitch decided to change things. v12 does not have any of the additional checks v13 does, so it's going to try claiming bonus points or claiming a drop, fail miserably, and then Twitch will invalidate your authentication token, leading to you getting the same error as the guy above next time you start the miner.

I know where the issue is, but now I need time. Lots of it. There's at least two ways this project can go forward with, but it's going to be a heavy decision to make, and thus I want to try some things before I do so. For now, you'll have to go back to mining the usual way. I'm sorry but I can't change this.

What do you mean by mining the usual way? Like, manually opening and watching channels?

DevilXD commented 1 year ago

@PockySweet Yes. There's no other way of getting the drops for now.

josephisticated commented 1 year ago

@DevilXD I don't know if it's rude to say this here but there is a somewhat working drops bot that crashes when it tries to collect points. I edited the code and removed that function and it seems it's in a working state. Would you like to check it so maybe you could find a solution to your problem? If you do please let me know and I will drop the link here or somewhere else you want.

DevilXD commented 1 year ago

@josephisticated I did mention above that the mining operation itself has not been impaired, because it doesn't interact with GQL - however the points claiming and drop claiming does. I don't see much reason to run this application vs just open a stream in your browser, if it's not going to claim anything automatically anyway. There is at least two other "miners" out there, that can do what this application does just as well, but it's going to keep trying to redeem points and drops, spamming Twitch with errors - and when that happens, usually developers bring out the ban hammer and block your IP from bothering their website.

These are only my words of precaution. I could just remove the autoclaiming feature in literally 10 minutes, but it'd just stop mining and get stuck on a single channel without being able to claim a drop, doing literally nothing. You're free to make a local copy and edit it to your own needs, if you know how - but again, I see no point if you can just open a web page to have the same functionality.

Besides, with the passport endpoint being protected with the same system as GQL, getting some auth information so that the watching would be even possible, is impossible now as well. There would need to be a separate system added for handling that instead. like hooking up to your browser and stealing the cookie from there - however, I always found that solution quite messy and easy to break, hence why I never even considered implementing it in my application.

I'm still in the process of re-thinking where exactly I can go from here, so please have patience. If you really cannot wait, you can try using other miners out there for now - not like I could stop you anyway. Just keep in mind what I said above, you're doing so at your own risk.

scryptio commented 1 year ago

@DevilXD Thanks for your reply, I totally missed the answer (because a few days prior it was working fine). But it seems twitch seems to actively fight bots (additionally to adblockers) now. There is a repo just trying to solve the integrity and the dev too advises not do use anything like that right now https://github.com/Kappador/twitch-integrity and one user guessed there that they implemented kasada antibot protection (mentioned elsewhere, too)

Twitch Channel Point Miner doesn't work either and this is one of the applications spamming the twitch API. But you can disable that.

If the mining is still working, you could think about deactivating the Claimer for now and just let it do the mining. The user needs to claim it himself, right, but doesn't need to keep full track of whether a stream started (for i.e. a 1day / few hours drop).

DevilXD commented 1 year ago

If the mining is still working, you could think about deactivating the Claimer for now and just let it do the mining. The user needs to claim it himself, right, but doesn't need to keep full track of whether a stream started (for i.e. a 1day / few hours drop).

Just "deactivating" the claiming isn't trivial, as I've already mentioned before. I could do it as a "for now" dev release just like I did earlier, in an attempt to go anywhere with this project. The thing is, the authorization endpoint is still kinda dead, so even that might refuse to work properly.

I've tried launching the miner today and it seemed to have passed the auth token verification - which might mean the passport endpoint isn't "quite dead" after all. I might actually try doing this and see how far I can salvage the current functionality - but testing anything at this point is quite risky, even for myself. This "kasada" thing, or whatever it's called, appears to be on-par (or greater) with the CAPTCHA solving challenge, which is why the CAPTCHA exception exists in the application. Trying to defeat this is really hard, but theorethically possible given enough time, dedication and JS knowledge, none of which I really have. I kept maintaining this project with small steps along the way, because I liked doing so to keep it fresh and going. Having to rewrite the majority of it because of something like this is quite painful in itself, I really liked what I've made so far.

Anyway, you can expect some kind of a new dev version in the nearest future. I'm currently quite busy IRL and what Twitch did has turned me off for this project quite a bit, but I'll still try to go somewhere with it, rather than simply abandoning it.

DevilXD commented 1 year ago

I've found this while doing research yesterday: https://github.com/WeaselStorm/ChannelPointCollector

Chrome and Firefox addon links are available in the repo description. Seems to work great if you can hang onto one of those infinite rerun streams, or simply know there's a long stream available, where you can just leave the tab open. Works by simulating user clicks via a simple side-loaded JS. Zero setup needed, just install and enjoy.

It gave me an idea in terms of making a use out of this knowledge, however this is still JS so outside of my field. It looks like there are ways to utilize Python in a browser extension, however this is still something I'd need time to research more, to be able to even tell if this would work as a miner.

Speaking of research, I figured I'd update you on my findings. So far I've looked into:

PyQt5/6 and it's QtWebEngine - found no documentation regarding being able to craft or alter custom requests, so while it does load the Twitch page, doing anything from there just isn't possible.
chromedriver.exe and geckodriver.exe as Webdrivers for Chrome and Firefox respectively. Since they take control of the browser you already have installed, and serve as an almost perfect browser-like environment that can deal with the pesky "challenge" JS, I saw it as quite a low hanging fruit. Unfortunately, bot detection scripts seem to always be one step ahead, and even after getting one of those "undetectable" Chrome drivers and passing two separate 3rd party bot tests (with a positive result), Twitch's integrity token still returns the "is_bad_bot" flag set to "true". Currently in the process of trying to see if adding some extensions and messing around with settings, so that they're not out-of-the-box default ones my Chrome starts with, could help mitigate the issue.
Will be investigating the possibility of writing a browser extension next.

Hope the linked extension above helps y'all :heart:

xmhyysd commented 1 year ago

383af8164e115dab25328243769b9e9

For me, compared with using browser plug-ins, it is not as good as TwitchDropsMiner, because network fluctuations often occur here, leading to network errors, making the browser no longer work. This makes the mining no longer efficient. So I look forward to the follow-up progress of the author. come on. Because it is a machine flip, there may be some reading comprehension errors. I'm sorry. 对于我来说，相较于使用浏览器插件，是不如TwitchDropsMiner的，因为在我这里经常会出现网络波动，导致网络错误，使浏览器不再工作。这就会使挖掘不再高效。所以期待作者的后续进展。加油！

DevilXD / TwitchDropsMiner

Application doesn't work anymore #40