Windows200000 / TwitchDropsMiner-updated

An app that allows you to AFK mine timed Twitch drops, with automatic drop claiming and channel switching.
MIT License
354 stars 19 forks source link

Twitch is down, retrying #172

Closed Windows200000 closed 2 months ago

Windows200000 commented 2 months ago

Are you experiencing this error? Please tell me!

I have it working for myself, where it tries more URLs and if it fails, doesn't get stuck in a retry loop. If other people have it, I will prioritize cleaning it up and make a commit.


Description

After a while, TDM gets stuck with the message Twitch is down, retrying in x seconds...

To Reproduce

  1. Start TDM

Expected behavior

it mines

Observed behavior

It retries, only very rarely actually mining

Screenshots

No response

Logs

No response

OS

Windows 11

Build

.exe

Version/Commit

v15.8.2

Additional context

Might be correlated with Summer Vibes Token Store: Drop 6 from World of Tanks.

Windows200000 commented 2 months ago

I have gotten to this being the issue, why exactly it repeatedly tried to connect to an invalid url? idk...

2024-08-17 23:50:16: Twitch is down, retrying in 118 seconds... Response: <ClientResponse(https://video-weaver.fra05.hls.ttvnw.net/v1/playlist/CvgEBj5v92UmilGcMzOfowkkBT1k9noaOcIXhhNj1MY_i1Po5veCSlX4o0Cxv_WitGB0MTtjvv1KlxDsLIY3sTnPwQsEa2zLttwaJT3A3MQs1um4gbV06AkU5bkF8MQgoxMQxwYlXy_dDmp6uaMkW1kd7vl1S7ZbYJ2IyGQSNJE7lDVfhwThK3pIv91Iit8uZqPg8R24xGN5I0uxfboPwCunNkQ6HqUs02Hxvzgs76IYiKazZd7xCK1kCLh4OBSkrijmgM7W4FYGCwJHHZ2eWQJDs-5qsx5LoEvmq8WhyE4ZK-F0V3DUjzL2j2FtVVAanaccI-SasuPqnthqEtfINHp1s2ba5FL2uhCvCoWBJ2sOphBXYV_U7iVMV3tkFuWKpg4VRvlhWPoCDe89sP9U-tl-V9XTD0_IA1IXeYewtpwJ_kLbOlmeFJXL3-Hj_fzktenDZ7hvq1Bo30S1MiMZALkXIO4zLuIQapkJA3uWUxMrYidatysOs19RSmOMAjAKthA7uqtLfqOAJ8AWYwpenqCHCYX3cjOrrBkKk2ZeXQbF1ive3amSbGhJrxE40kA1wmgsIb81LQLGZUN9YpAa1bpdxlsJXbpoaMS0EVzuOCaE2PG-gIMOhS9IjM0KJd1VwTYILS3ynilsw1hw9XTz6sI_xpXbp0C2aWn138QswMko80ffrMzr2fd58sSOjz6l2WJ4u8w4TWzQvRCjQ-WXCHGKrJZVAeGjLwiRHCYwRlxa39RPGh0FR6W_DZlWHnG4dSZZfLMgFtxJuulJkqNtasDCkdh4Nh830IEBq89ytV099wptpAUwYhpME3JUUpiMCnTJ5AxolWFUwAsaDAHLRu3TyL490LHLNyABKglldS13ZXN0LTIwlQo.m3u8) [500 Internal Server Error]> <CIMultiDictProxy('Content-Type': 'text/plain; charset=utf-8', 'Vary': 'Accept-Encoding', 'x-content-type-options': 'nosniff', 'x-error-message': 'CLUSTERDOWN Hash slot not served', 'Date': 'Sat, 17 Aug 2024 21:50:14 GMT', 'Content-Length': '22', 'Access-Control-Allow-Origin': '*')>

Windows200000 commented 2 months ago

image So the Twitch site gets a list of links to m3u8 playlists that work, while TDM gets a list where instead of the m3u8 playlist a 500 Internal Server Error gets returned.

Windows200000 commented 2 months ago

I finishes mining Summer Vibes Token Store: Drop 6 from World of Tanks and the issue persists. It also looks like not all links have to be broken in the response TDM gets.

Windows200000 commented 2 months ago

The issue seems to be it gets one bad url and then just retries that one again and again instead of trying to get another one. I'll look deeper tmrw.

Windows200000 commented 2 months ago

Of 60 requests, 57 went through and 3 failed with all 4 different broadcast qualities. Maybe I made a mistake yesterday when I thought I figured out that it's not always all broken or all working.

update: 3/207 successful ones did end up working with the 2nd link. I must've gotten really lucky noticing that yesterday :D

Windows200000 commented 2 months ago

So immediately retrying the send_watch function makes it work. This means we can probably just do that after the first fail.

A pretty serious issue is that TDM doesn't stop counting down once it fails. This is a really difficult issue to solve, because a lot of the code still relies on minute intervals. I hacked the new 20s on top by just not advancing a minute if there is still seconds remaining. That means nothing checks if 2/3 of the requests even succeeded, and it's just assumed they did.

Windows200000 commented 2 months ago

here you can check what I currently have: https://github.com/Windows200000/TwitchDropsMiner-updated/compare/in-dev...Windows200000:TwitchDropsMiner-updated:issue-172-ttv-down-retrying

Skay100 commented 2 months ago

For me, it completely freezes, once it unfreezes I do get the error, but any action on the program freezes it again, before the error everything was completely fine.

Windows200000 commented 2 months ago

Compile from the fork in my above comment. This seems to be pretty rare. Or take this exe: Twitch Drops Miner (by DevilXD).json

notNSANE commented 2 months ago

yes, im having this issue and progress is being reset too, very annoying

Skay100 commented 2 months ago

Sadly your exe doesn't work for me too. 3/50 Websockets connected, at first it gets channel list, ends up breaking on rewards that might be a subscription reward, goes back to "gathering channels" and can't connect to Twitch again. (which freezes the app constantly)

Windows200000 commented 2 months ago

@Skay100 @notNSANE

Sadly your exe doesn't work for me too. 3/50 Websockets connected, at first it gets channel list, ends up breaking on rewards that might be a subscription reward, goes back to "gathering channels" and can't connect to Twitch again. (which freezes the app constantly)

The subscription reward is ignored. You have to rename the Json I sent 3 messages above to an exe. GitHub doesn't allow executables in comments. Try clearing your cookies file and logging in again too.

Sparh4wk commented 2 months ago

@Skay100 @notNSANE

Sadly your exe doesn't work for me too. 3/50 Websockets connected, at first it gets channel list, ends up breaking on rewards that might be a subscription reward, goes back to "gathering channels" and can't connect to Twitch again. (which freezes the app constantly)

The subscription reward is ignored. You have to rename the Json I sent 3 messages above to an exe. GitHub doesn't allow executables in comments. Try clearing your cookies file and logging in again too.

sadly its the same... Snímek obrazovky 2024-08-22 171808 ad

JourneyOver commented 2 months ago

Compile from the fork in my above comment. This seems to be pretty rare. Or take this exe: Twitch Drops Miner (by DevilXD).json

This seems to be working for me, it did freeze like others mentioned for a minute at most while it was gathering channels but then it eventually unfroze, threw a connection error for a slight bit and then it started working just fine.

I do get random:

13:04:06: ERROR:  NOT ERROR: Successfully watched after 0 retries.
13:04:27: ERROR:  NOT ERROR: Successfully watched after 0 retries.
13:04:48: ERROR:  NOT ERROR: Successfully watched after 0 retries.

things coming up in the output section a ton but it at least seems to be progressing on twitch drops as well.

Screenshot 2024-08-22 130814

Windows200000 commented 2 months ago

yup, the error not error is what I used to try and figure out what;s going on, you can ignore that. The freeze doesn't affect me, I guess it's a separate issue?

JourneyOver commented 2 months ago

the error not error is what I used to try and figure out what;s going on, you can ignore that

Yea I kind of figured that it could be ignored for the most part.

The freeze doesn't affect me, I guess it's a separate issue?

Possibly, I only had it freeze the one time during the gathering channels bit and it was a very short amount of time that the window was frozen, other than that everything seems to be running fine for me though so far for the past 3 hours that I've been running the version posted above.

JourneyOver commented 2 months ago

Ooop I ran into it now just throwing twitch is down retrying error after I had a intermitten downtime of my internet, I ended up shutting down TDM for the time being that my internet was down and once my internet was back up I re-opened TDM again and it has just been stuck on cannot connect to twitch, retrying in X seconds ever since.

I did try the original version of TDM from DevilXD and it seems to be working perfectly on connecting and all that, so it def seems to be something to do with this specific fork..

luisf371 commented 2 months ago

Using the updated .exe provided above, im still receiving the error. Let me know if there's any logging i can provide to assist. image

zelda0079 commented 2 months ago

Delete the cookies.jar.

JourneyOver commented 2 months ago

Delete the cookies.jar.

That doesn't really make any difference on my end.

zelda0079 commented 2 months ago

Delete the cookies.jar.

That doesn't really make any difference on my end.

This is temporary fix when it is happening.

Windows200000 commented 2 months ago

I merged the branch into in-dev. I will be running it on my end as well and if it works I'll make a release. I'm still really busy, unfortunately.

Download here.

JourneyOver commented 2 months ago

So far everything has been running perfectly on my end \^^

T4bletopG4mes commented 2 months ago

Updated to the dev build...problem is back. Its 100% twitch's fault but also TDM seems to freeze when gathering campaigns and channels.

JourneyOver commented 2 months ago

Have no issue with TDM freezing still, but did run into the Twitch is down, retrying in x seconds... loop again today. Only solution was to delete the cookies.jar file and relog in again to create a new cookies.jar file after doing that everything started working again.

zelda0079 commented 2 months ago

Have no issue with TDM freezing still, but did run into the Twitch is down, retrying in x seconds... loop again today. Only solution was to delete the cookies.jar file and relog in again to create a new cookies.jar file after doing that everything started working again.

You can copy cookies.jar(4kb) as a backup, you don't need to log in again.

Windows200000 commented 2 months ago

Weird, that it still happens, because it's been working for me over a couple of days with at least a dozen of mined hours.

I guess I'll keep this open for now, but if it happens, please use v15.9.0, and upload the full log, ran with -vvv --debug-gql --debug-ws --log.

(You have to run it via CMD, a shortcut, Win + R or similar to be able to add the arguments)

Skay100 commented 2 months ago

It started working today for me (I was clearing cookies every day), I was experiencing every single issue known to mankind including miner slowing down my PC to absurd degree, today after cookies/cache clear it works flawlessly again (from your exe provided in earlier comments). It's more responsive than ever.

JourneyOver commented 2 months ago

Weird, that it still happens, because it's been working for me over a couple of days with at least a dozen of mined hours.

It had been working fine for me as well for the past couple of days since switching over to using the in-dev code, so far this has been the second time only that it has happened, but the thing that was weird about this one was that it was still mining (albiet it kept returning the same progress on an item drop but it was still actually progressing on other items when I viewed my twitch inventory) while throwing the connection error like how it shows in the following snippet from my logs:

Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
CALL: Successfully watched after 0 retries.
CALL: returned watch, succeeded: True, repeat_new: False
CALL: No drop update from the websocket received
Progress: 120/120 - Campaign(Call of Duty: Black Ops 6, COD Beta 2024, 1/4)
CALL: Drop progress from active search: Calling Card  (Call of Duty: Black Ops 6, 120/120)
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...
Cannot connect to Twitch, retrying in 180 seconds...

Was stuck doing something along those lines on repeat even though it was still actually progressing on other items in the same campaign until I did the whole shutdown, deleted the cookies file and restarted everything and then everything started working fine again and been since.

You can copy cookies.jar(4kb) as a backup, you don't need to log in again.

good to know, I'll try that next time.

JourneyOver commented 2 months ago

I've encountered the issue again where it gets stuck on the cannot connect to Twitch message and remains stuck on a single drop. Unfortunately, I wasn't running with debug flags enabled at the time, but I am now. This time, after shutting down the container (I'm running this through Docker btw), I restarted it with the -vvv --debug-gql --debug-ws --log flags, and it started working again without needing to delete or replace the cookies.jar file.

Before restarting with the debug flags, I downloaded the logs from my container and noticed that the following error was being repeatedly logged every so often about the same time the connection issue started:

ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.12/http/cookies.py:295: ResourceWarning: unclosed <socket.socket fd=35, family=2, type=1, proto=6, laddr=('172.18.0.25', 46066), raddr=('151.101.194.214', 443)>

(Note: The raddr value varies with each occurrence.)

I hadn't seen the resource warning coming up any before hitting the connection problem, though I suspect it was likely happening when I previously said something about the connection issue happening as well, but I just didn't catch it then.

I've got the debug information running now, so if the issue happens again, I'll upload the log from my container. @Windows200000, if it does happen, is there any sensitive information I should remove from the log beforehand?

Lastly, are these two lines here responsible for controlling the Cannot connect to Twitch, retrying in X seconds message, or is it located elsewhere? When searching for connection-related code, I only found translation-related stuff and these few lines.

Windows200000 commented 2 months ago

@Windows200000, if it does happen, is there any sensitive information I should remove from the log beforehand?

I didn't add any, but DevilXD might have. I know none of these are sensitive: {'operationName': 'DropCampaignDetails', 'extensions': {'persistedQuery': {'version': 1, 'sha256Hash': 'e7acdecb05429a62f5984bdcb27ee938ae20543579bf73c3ae44e7c822bc4f54'}}, 'variables': {'channelLogin': '209080157', 'dropID': '969c83d3-6374-11ef-ac41-0a58a9feac02'}}

I just asked him and will reply again once I get an answer.

This-Person commented 2 months ago

I never had this issue previously but I downloaded 15.9.0 because I like to make sure I'm up to date. After that I constantly got the retrying error. I downloaded 15.8.2 again but the error persisted with that version too even though it didn't happen with that version before. Both versions of the app felt very sluggish and froze a lot when trying to access inventory/settings.

I downloaded DevilXD's latest version and dropped it into the same folder and that is working fine. Very strange issue.

I've made a log I can send over if you need 👍

Edit:

Miner error

This is what it looks like when I run 15.9.0 now. CPU usage is fairly high when it's running, using about 10% of my 16 core CPU.

JourneyOver commented 2 months ago

@This-Person can you try running the -vvv --debug-gql --debug-ws --log flags as mentioned in https://github.com/Windows200000/TwitchDropsMiner-updated/issues/172#issuecomment-2322318548 and see if it produces anything while it's throwing to the cannot connect to twitch error? That's the type of log he is trying to get right now is with those flags.

JourneyOver commented 2 months ago

It is strange thing though, as even when I run the application side of things instead of going and running through docker, I haven't experienced the sluggish/freeze issue that others have mentioned any other than the very first time I ran the in-dev version that was mentioned way above before 15.9.0 was released, any other time I've ran the application I've not experienced it since, even though I have still experienced the connection error several times now.

Still waiting for the error to happen again on my end but so far haven't seen it again since I started back up with the debug flags 6 hours ago >.<

This-Person commented 2 months ago

@This-Person can you try running the -vvv --debug-gql --debug-ws --log flags as mentioned in #172 (comment) and see if it produces anything while it's throwing to the cannot connect to twitch error? That's the type of log he is trying to get right now is with those flags.

I won't lie I have no idea what I'm looking at but I can't see any obvious errors. Lots of messages like this in a row for different channel names:

2024-08-31 09:38:02.887: DEBUG: GQL Request: {'operationName': 'VideoPlayerStreamInfoOverlayChannel', 'extensions': {'persistedQuery': {'version': 1, 'sha256Hash': 'HASH'}}, 'variables': {'channel': 'zigueira'}}

Then after those messages stop a lot of messages like this in a row for different channel names:

2024-08-31 09:38:31.382: DEBUG: GQL Response: {'data': {'user': {'id': '172985691', 'profileURL': 'https://www.twitch.tv/legionlooterana', 'displayName': 'LegionLooterana', 'login': 'legionlooterana', 'profileImageURL': 'https://static-cdn.jtvnw.net/jtv_user_pictures/07da15c9-084f-4eea-a8d6-e3f96f405d1d-profile_image-150x150.png', 'broadcastSettings': {'id': '172985691', 'title': 'Jugamos y opinamos sobre Call of duty Black Ops 6 #Playstation5 #Directo #MasterChef', 'game': {'id': '1672324422', 'displayName': 'Call of Duty: Black Ops 6', 'name': 'Call of Duty: Black Ops 6', '__typename': 'Game'}, '__typename': 'BroadcastSettings'}, 'stream': None, '__typename': 'User'}}, 'extensions': {'durationMilliseconds': 57, 'operationName': 'VideoPlayerStreamInfoOverlayChannel', 'requestID': 'REQUESTID'}}

I've removed the requestID and sha256Hash just in case they're identifiable.

As I say without changing anything at all, deleting the 15.9.0 exe and dropping DevilXD's latest exe into the same folder instead makes everything work again. No idea what's causing it.

Edit: Unsure if it's pertinent but as soon as I run 15.9.0 I get this message everytime: 10:31:51: Required_minutes for "Law and Momento Trinkets" from "Star Wars Outlaws" is 0. This could be due to a subscription requirement, tracked in Issue #101. Take a look at the drop, but this can likely be ignored.

I don't have Star Wars Outlaws in my list of games to mine so not sure what it's mentioned. Although checking the logs, despite not having the game in my list it mentions some streamers streaming the game? An example is below:

2024-08-31 09:38:49.178: DEBUG: GQL Response: {'data': {'user': {'id': '137699110', 'profileURL': 'https://www.twitch.tv/moska1m', 'displayName': 'Moska1m', 'login': 'moska1m', 'profileImageURL': 'https://static-cdn.jtvnw.net/jtv_user_pictures/5de62d68-b7dc-4430-8698-7b1f9050d1a2-profile_image-150x150.png', 'broadcastSettings': {'id': '137699110', 'title': 'Guerra nas Estrelas Foras da Lei - #StarWarsOutlaws #ad', 'game': {'id': '780037970', 'displayName': 'Star Wars Outlaws', 'name': 'Star Wars Outlaws', '__typename': 'Game'}, '__typename': 'BroadcastSettings'}, 'stream': None, '__typename': 'User'}}, 'extensions': {'durationMilliseconds': 59, 'operationName': 'VideoPlayerStreamInfoOverlayChannel', 'requestID': 'REQUESTID'}}

Windows200000 commented 2 months ago

requestID and sha256Hash

The hash is hard-coded for the persistent query, and the requestID is likely invalid after a very short time.

However, DevilXD said, that at some point the authorization token might get logged, which would be a pretty huge issue, that's pretty much the same thing as malicious extensions, that steal your cookies and can gain access to your sessions (logins).

I'm gonna let mine run for a bit and search for any weird stuff.

Windows200000 commented 2 months ago

Actually, yeah, there's the auth_token in there, which isn't great.

Try using only -vvv, that won't allow me to see exactly what's going on, but it should ensure there is no "raw" requests and responses, that contain most of the stuff your PC and Twitch needs to communicate.

Sparh4wk commented 2 months ago

adding my logfile if someone is interested log.txt

Windows200000 commented 2 months ago

@Sparh4wk remove it, it contains your auth token. Sorry for not clarifying that, I wasn't aware myself. I'll add you to a GH support ticket I have about this.

Windows200000 commented 2 months ago

@Sparh4wk

They're all deleted now by GitHub support, including yours.

Sparh4wk commented 2 months ago

@Sparh4wk

They're all deleted now by GitHub support, including yours.

Thanks. I was afk till now

This-Person commented 2 months ago

I can confirm that after 4 or 5 minutes the CPU usage goes down, the 'cannot connect to Twitch, retrying' error disappears, the app stops freezing/being sluggish and the app starts mining normally. I've let it run for 10 mins to get a wee -vvv log which I'll attach here, hopefully it helps! If you need a more in depth log let me know and I can e-mail it over or something 👍

log-1.txt

Edit: At 22:28 I pushed reload in the Settings to enable autostart and the CPU usage spiked again and the error returned for another few minutes if something looks weird in the log.

This-Person commented 2 months ago

Apologies for the double post but wanted to give a small update. I switched my laptop on this morning and the 15.9.0 app just auto launched normally. No high CPU usage or error messages, it just checked the inventory then got straight into mining. Very strange!

JourneyOver commented 2 months ago

Unsure of why this was closed fully as it is still an on-going issue. I do seem to notice a huge spike in CPU/Ram that others are talking about while it's running into the twitch is down retrying error, upon reloading the program (or in my case the container) the usage does go back to normal until it goes and hits the problem again.

I did switch back over to the -vvv as suggested recently as well during all this..

twitchdropsminer-2024-09-03T00-48-45.log

Windows200000 commented 2 months ago

Unsure of why this was closed fully as it is still an on-going issue.

I closed it, because I thought it was a separate issue, since it has been completely resolved for me.

I do seem to notice a huge spike in CPU/Ram that others are talking about while it's running into the twitch is down retrying error, upon reloading the program (or in my case the container) the usage does go back to normal until it goes and hits the problem again.

Thanks a ton for the log! It seems like it is now retrying every 2 seconds, which is not good. But it is quite likely caused by the container. Python is really not the right programming language for this, so the exe is janky solution that generates the .py fiels in a temp directory before running them. If you look ath the forks of the upstream, there should be another one, that is meant to be run in docker. I strongly suggest you use that.

From README.md:

TDM is not intended for/as:

  • Mining channel points - again, it's about the drops: only. The current points you're getting are a byproduct of getting the drops, not the main goal of it.
  • Mining anything else besides Twitch drops - no, I won't be adding support for a random 3rd party site that also happens to rely on watching Twitch streams.
  • Unattended operation: worst case scenario, it'll stop working and you'll hopefully notice that at some point. Hopefully.
  • 100% uptime application, due to the underlying nature of it, expect fatal errors to happen every so often.
  • Being hosted on a remote server as a 24/7 miner.
  • Being used with more than one managed account.
  • Mining campaigns the managed account isn't linked to.
JourneyOver commented 2 months ago

If you look ath the forks of the upstream, there should be another one, that is meant to be run in docker. I strongly suggest you use that.

@Windows200000 If you're referring to @Valentin-Metz fork, I forked his version recently (couple days back (was the 18th or 19th of last month)) to integrate updates from this repo (which he pulls code from this repo and is pretty much the base of his code) more quickly into my container and to make some adjustments and adding my own modifications to the health checker bits. I'm not using the .exe version within the container; instead, it runs the command python main.py -vvv, which directly executes the Python code. I guess I should of mentioned that when I made my last comment that I wasn't running the exe program directly in the container..

I'm mentioning the retry connection error issue here because (other than this issue being a direct corrolation to what I'm experiencing and I'm sure others are still as well), during my testing of the .exe version from this repository (outside of Docker), I encountered the same connection retry problem during all the times I've commented in this issue. The last time I ran the .exe version on my main machine was last night, shortly after uploading the log from my container because you had closed this issue and I wanted to test to see if there was something different I wasn't seeing. About 2-3 hours after starting it, I ran into a connection retry error. Interestingly, I didn't seem to experience connection retry errors any while testing the default version of TDM (DevilXD's version) however as I had to test it last night due to dealing with a different issue related to a specific drop, which I'll mention below.

Unfortunately, I don't recall the exact error that forced me to switch to Devil's version of TDM, as I was too tired and forgot to write it down. However, last night, I encountered a specific issue with the Once Human Twitch drop, which seems to only affect that particular drop. While farming the Once Human Twitch drop, the program would hit a GQLException every few minutes, regardless of the stream, and stop farming the drop. The program remained functional, but it would just sit idle until I either switched channels or restarted the program. I eventually had to switch to the default version (DevilXD's version), which didn't seem to run into this problem any. Testing on both my main machine with the .exe version from this repository and within my Docker container using the Python code consistently reproduced the issue.

Valentin-Metz commented 2 months ago

@Windows200000 do you plan on merging your fork with devils upstream? Is there any reason this has not been done already (as @DevilXD seems to be active again)? I'm also experiencing the issues @JourneyOver mentioned, so I wonder whether I should put in the effort to port back to the original upstream.

Windows200000 commented 2 months ago

@Windows200000 do you plan on merging your fork with devils upstream? Is there any reason this has not been done already (as @DevilXD seems to be active again)? I'm also experiencing the issues @JourneyOver mentioned, so I wonder whether I should put in the effort to port back to the original upstream.

I am looking to merge it. The issue is that DevilXD doesn't like some parts, like prioritize by ending soonest which I didn't write, and the dark mode implementation, which he would rather be implemented more cleanly, even if you then potentially need to restart to apply changes.

I'll work on that once I have some time, hopefully relatively soon.

DevilXD commented 2 months ago

Since I've been mentioned here, and read that y'all have an issue with the "Gathering channels..." phase of the reload process, I figured I'd shed some light on this.

The reason it freezes and gets into the "Can't connect to Twitch" loop, is because this fork still tries to set all ACL channels online at once on reload:

https://github.com/Windows200000/TwitchDropsMiner-updated/blob/42fe40423fcf77930efb962792f45d12fc195882/twitch.py#L913-L917

..., while I disabled that almost a month ago: https://github.com/DevilXD/TwitchDropsMiner/commit/f705cc1527e806bb979df9e0dbd077d93728160c

The cause of this is Twitch implementing a rate-limit on GQL requests, that'll return "Unauthorized" if there's lots of GQL requests being sent at once, which is exactly what happens when the channel status is loaded in the code above, and there's a lot of channels to load. Ref issue on my repo: https://github.com/DevilXD/TwitchDropsMiner/issues/526

Disabling this piece of code means that the miner may throw the "No available channels to watch. Waiting for an ONLINE channel..." message into the output window once the reload finishes, but it can be safely ignored. The channels are still gathered, the websocket still subscribes to channel status updates, and Twitch will still send channel status updates to the miner, including the "viewcount" event, that is sent repeatedly every so often (every like 30s), which main purpose is to update the viewer count for a stream - btw that's how you can see the viewer count updated in real time under a stream on the website.

Once the event arrives, the miner processes it into updating the channel's entry on the Channels list. But if the channel's status happens to be OFFLINE, the miner realizes it has an outdated channel state, and sets it online: https://github.com/Windows200000/TwitchDropsMiner-updated/blob/42fe40423fcf77930efb962792f45d12fc195882/twitch.py#L1233-L1245

Setting a channel online has a 2 minutes of a delay added to it, because the usual way a channel is set ONLINE, is by receiving the "stream-up" event. and these events are usually sent way before the stream actually starts. These 2 minutes are really generous, as the actual delay roughly matches the usual delay you get from watching a stream, if you've ever watched someone stream and respond to chat messages, it's more like 30 seconds max instead of the 2 minutes, but I've never really bothered min-max optimizing this - some delay was needed, 2 minutes worked, I just left it at that.

So shortly after a reload, you'll see the miner switch to the IDLE state, but shortly afterwards, some channels on the channel list will change their status from OFFLINE ❌ to OFFLINE ⌛, indicating that they're about to be checked if the stream is actually ONLINE, as an event indicating so was received. 2 minutes later, a channel will update it's status to ONLINE ✔, triggering a switch event, and causing the miner to start mining as usual.

Ref issues for more info on the above: https://github.com/DevilXD/TwitchDropsMiner/issues/537, https://github.com/DevilXD/TwitchDropsMiner/issues/541, https://github.com/DevilXD/TwitchDropsMiner/issues/551.

That's how the current master branch of mine works, until I can rewrite the commented out code, as it can be optimized to not send up to ~200 GQL requests at once, which is what happens if you have several games added to the Priority list, and they all together happen to reach the channels limit of the miner - which is also why this issue happens only to some people, and reproducing it is "unreliable".

JourneyOver commented 2 months ago

@Windows200000 lol did you have a stroke while writing that last message xD

@DevilXD hmm, I'll push that commit into my repo and keep an eye on things to see if it makes things better or not.