"Error: 'NoneType' object is not subscriptable" on bulk_check_online

[matarife123]

[DevilXD]

Hello,

I've seen this before, and it still makes no sense. What games have you added to your priority list?

[matarife123]

I didn't make any changes, I only have 4 or 5

[DevilXD]

Yes, but you also have the priority mode set to "Ending soonest", which causes it to collect many more campaigns than those that are on the priority list.

Please run the miner with the --dump argument, and upload your dump.dat file here. I'll need to take a closer look at it.

[matarife123]

The only thing I know is that they added exclusive drops for people who gift subscriptions, maybe this information will be useful.

[matarife123]

Sí, pero también tienes el modo de prioridad configurado en "Terminar lo antes posible", lo que hace que recopile muchas más campañas que las que están en la lista de prioridades.

Ejecute el minero con el --dumpargumento y cargue su dump.datarchivo aquí. Tendré que analizarlo más detenidamente.

Sorry, I'm a basic user, I don't understand programming.

[DevilXD]

The sub-based drops are already ignored in the miner.

Running a program with arguments has nothing to do with programming, it's quite basic knowledge of how to use your computer, although also not something you do every day I guess. I've already written an instruction for this once today, so you can just refer to that here: https://github.com/DevilXD/TwitchDropsMiner/issues/557#issuecomment-2351759325

[matarife123]

dump.zip

[DevilXD]

Does the issue go away if you switch the Priority mode to "Priority list only"?

[matarife123]

¿El problema desaparece si cambia el modo de prioridad a "Solo lista de prioridades"?

yes

[DevilXD]

It seems the GQL rate limit may still be the problem here then. I've thought that the miner can work without it for now, but clearly that's not the case.

I'm temporarily disabling the status refresh (https://github.com/DevilXD/TwitchDropsMiner/commit/8fa0669c9f44270b821a534104c6d849470fe048) until I can implement a rate limiter for GQL.

[matarife123]

thanks, work now :D

[DevilXD]

The rate limiter has been implemented. Please check again with the latest master.

[JourneyOver]

Edit: Nvm it seems the error is still happening:

[JourneyOver]

dump.dat.txt if you need it. I added the .txt extention to be able to upload it here, you can remove the extension if you need to.

The error seems to only be happening if "Ending Soonest" or "Low Availability" is selected. if "Priority list only" is selected the error doesn't seem to happen.

[JourneyOver]

Also something I've noticed recently is an issue with the "Ending Soonest" and "Low Availability" options that I am going to create a new issue for.

[DevilXD]

It's most likely because of Ravendawn, as it has over 1000 ACL channels, resulting in 50 requests needing to be sent for this game/campaign alone. I'd suggest putting it on the Exclude list until this issue resolves.

I'm not entirely sure what the exact rate limit is, but the current 20/1 seemed very reasonable already. I need to do more research on this topic over the next few days.

[DevilXD]

I've increased the rate limit 4-fold, to a point it may be quite noticeable during the "Gathering channels" phase now. The 1000 channels of Ravendawn should take about 10 seconds to fetch, the other games combined can have up to 1500, so it may take a while. It'd be best to do a before-and-after test for this.

Does it work with Ravendawn excluded? Does it work with Ravendawn not excluded? Before updating? After updating?

What about Ravendawn being on the Priority list as the only game, and priority mode set to "Priority list only"?

[JourneyOver]

previously before the newest commit it would crash out unless priority mode was set to priority list only didn't matter where on the list in my list of games, was same deal with it being the only option in my priority list as well and also excluding unless choosing priority list only

with the new code and it set to priority list only seems to be working perfectly fine with the amount of games I have in my list and having ravendawn anywhere in the list.

with the new code and it set to Ending Soonest or Low Availability it crashes with the amount of games I have in my list doesn't matter where at in the list.

Adding ravendawn to my exclude list instead still produces the same crash when set to Ending Soonest or Low Availability, does not produce a crash however when set to priority list only

Doesn't crash when ravendawn is the only game in my list with priority list only selected, does crash if switched over to either of the other two however with just ravendawn in the priority list.

This seems to be the error that is producing (which seems to be about the same as before):

05:46:18: Fatal error encountered:
05:46:18: 
05:46:18: Traceback (most recent call last):
05:46:18:   File "main.py", line 155, in main
05:46:18:     exit_status = 0
05:46:18:   File "twitch.py", line 587, in run
05:46:18:     while True:
05:46:18:   File "twitch.py", line 747, in _run
05:46:18:     if campaign.allowed_channels:
05:46:18:   File "twitch.py", line 1625, in bulk_check_online
05:46:18:     {"input": {"channelID": str(channel_id), "claimID": claim_id}}
05:46:18: TypeError: 'NoneType' object is not subscriptable
05:46:18: 
05:46:18: Exiting...
05:46:19: 
05:46:19: Application Terminated.
05:46:19: Close the window to exit the application.

---

Edit: It'll be about ~12ish hours before I'm able to get back to anymore comments or trying any more new commits, as I need some sleep before I head off to work xD but anything new pops up I'll test it later when I can.

[DevilXD]

I'll try reproducing this on my side, once I get back from work.

[JourneyOver]

So one thing I can at least confirm so far with a tiny bit more testing on this is that if ravendawn is the only thing your twitch account is linked to (I had set up a new twitch account to test this), it doesn't throw an error at all on any of the priority modes. The crashes seem to only happen when you start having a decent amount of games linked to your twitch account and not using the priority list only mode(unsure of how many exactly as I don't feel like sitting here and switching account links back and fourth and such). but it'll at least give a hint on hopefully debugging/reproducing on your end.

[DevilXD]

It's a matter of determining if it's really a GQL rate limit that's being at play here, or yet another one of those Twitch bugs that me and others happen to discover sometimes. The error points out that Twitch, when asked about channel information, has provided an empty response, just like the channel wouldn't exist. Since that makes no sense, it can be either a Twitch bug returning an empty object, or Twitch returning an empty object because it detected lots of GQL requests coming in too fast, and decided throwing a wrench into the gears is it's best option.

I'm currently working on #563, but if you'd want to help me test this, it'd be best to narrow it down to a single game, single campaign, or a single channel, that causes this issue. If I can repro it on my side, I can try fixing it, otherwise it's pretty much a cat-n-mouse game of me chasing the errors others are getting, asking them to test things, and hoping for the best.

[JourneyOver]

So I did a tiny bit of debugging on my side just to see what exactly was returning the "None" object

went in and added:

                # Check if the response_json or its key is None
                if response_json is None:
                    print("Received None response in response_list")
                    continue

                if "data" not in response_json or response_json["data"] is None:
                    print(f"response_json does not contain 'data' or it's None: {response_json}")
                    continue

                if "user" not in response_json["data"] or response_json["data"]["user"] is None:
                    print(f"'user' not in 'data' or 'user' is None: {response_json['data']}")
                    continue

                channel_data: JsonType = response_json["data"]["user"]
                if channel_data is None or "id" not in channel_data:
                    print(f"channel_data is None or missing 'id': {channel_data}")
                    continue

under for response_json in response_list: in the async def bulk_check_online(self, channels: abc.Iterable[Channel]): code in twitch.py

and from what I can tell so far is that at some point during the response it returns a "None" USERobject in the response data; the exact response I got was 'user' not in 'data' or 'user' is None: {'user': None}, the funny thing is that since I added that code it does seem to bypass things and goes straight to farming ravendawn like nothing happened after it throws the error message. Currently working on trying to narrow it down some more if I possibly can.

---

Did another debug with a tiny bit more changed:

diff --git a/twitch.py b/twitch.py
index 42b6a79..02a9dd4 100644
--- a/twitch.py
+++ b/twitch.py
@@ -1615,14 +1615,28 @@ class Twitch:
             # shortcut for nothing to process
             # NOTE: Have to do this here, becase "channels" can be any iterable
             return
-        for coro in asyncio.as_completed([
-            self.gql_request(stream_gql_chunk)
-            for stream_gql_chunk in chunk(stream_gql_ops, 20)
-        ]):
-            response_list: list[JsonType] = await coro
-            for response_json in response_list:
-                channel_data: JsonType = response_json["data"]["user"]
-                acl_streams_map[int(channel_data["id"])] = channel_data
+        for stream_gql_chunk in chunk(stream_gql_ops, 20):
+            for coro in asyncio.as_completed([
+                self.gql_request(stream_gql_chunk)
+            ]):
+                response_list: list[JsonType] = await coro
+                for response_json in response_list:
+                    # Log the specific GQL operation related to the response
+                    related_operation = stream_gql_chunk[response_list.index(response_json)]
+                    if response_json is None:
+                        print(f"Received None response for operation: {related_operation}")
+                        continue
+                    if "data" not in response_json or response_json["data"] is None:
+                        print(f"response_json missing 'data' or 'data' is None for operation: {related_operation}")
+                        continue
+                    if "user" not in response_json["data"] or response_json["data"]["user"] is None:
+                        print(f"'user' not in 'data' or 'user' is None: {response_json['data']} for operation: {related_operation}")
+                        continue
+                    channel_data: JsonType = response_json["data"]["user"]
+                    if channel_data is None or "id" not in channel_data:
+                        print(f"channel_data is None or missing 'id': {channel_data} for operation: {related_operation}")
+                        continue
+                    acl_streams_map[int(channel_data["id"])] = channel_data
         # for all channels with an active stream, check the available drops as well
         acl_available_drops_map: dict[int, list[JsonType]] = {}
         available_gql_ops: list[GQLOperation] = [

and so far everytime it returns the same single response:

'user' not in 'data' or 'user' is None: {'user': None} for operation: {'operationName': 'VideoPlayerStreamInfoOverlayChannel', 'extensions': {'persistedQuery': {'version': 1, 'sha256Hash': '[redactedasidkifitssafetshare]'}}, 'variables': {'channel': 'bizcochitogaming'}}

I redacted the sha256Hash for now as idk if it is safe to share or not exactly, but if it is then let me know and i'll post it as well.. but so far that has been the same consecutive channel to come up so far everytime I restart TDM with the debug code.

[DevilXD]

A reliable repro is all I need to fix this.

the funny thing is that since I added that code it does seem to bypass things and goes straight to farming ravendawn like nothing happened

Whatever causes the error, it's effects don't seem to last long, so it's possible. Still, I'd like to remind you that the previous client ID also had issues like that at some point, and like three days later, Twitch cut off the client from GQL entirely (see #526). I'd like to avoid hammering Twitch with error requests if possible, so please try not to "just let it loop back and get there anyway", because that's not the point. The miner has to coexist with Twitch, not causing them to kill it.

[JourneyOver]

Whatever causes the error, it's effects don't seem to last long, so it's possible. Still, I'd like to remind you that the previous client ID also had issues like that at some point, and like three days later, Twitch cut off the client from GQL entirely. I'd like to avoid hammering Twitch with error requests if possible, so please try not to "just let it loop back and get there anyway", because that's not the point. The miner has to coexist with Twitch, not causing them to kill it.

Yes I know, I was just trying to debug things to figure it out what exactly was causing the issue and then I was going to remove the code, so far it seem to be to do with that specific channel (the one I posted about in my edit) returning a None user object though from what I can tell. If I try to go to that specific channel on twitch it does come up with This channel is currently unavailable due to a violation of Twitch's Community Guidelines or Terms of Service. so I'm wondering if it's picking up channels that have been banned/suspended and that is what is causing it specifically? but I'm honestly not 100% sure, all I know is so far I've been able to reproduce it on my main twitch account which has a decent amount of games link to it.

if there is any code changes you could possibly think of that I might be able to try on my end to help lemme know and I will try them.

[DevilXD]

Hmm, you might be onto something here then. Is it just the bizcochitogaming channel causing this?

[JourneyOver]

So far everytime I've ran the code that I had pasted above in the patch edit, it has always returned that specific channel in the response and no others, even if I let it run for a bit.

[DevilXD]

I can see the site shows the same behavior on the web version. This pretty much confirms it that this is just how Twitch reports suspended channels then:

If this is the case, then it'd be just a matter of, well, skipping processing of channels like that, and treating them as OFFLINE. I can try putting this together, but I'll need to do it tomorrow, because it's night time for me.

For now, it should be safe to simply let it skip that single channel. During debugging, I've found some other issues that need addressing as well, but I'll work on it tomorrow too.

[JourneyOver]

Sounds good! At least we seem to have narrowed down this issue possibly. I'll keep an eye out for the commits tomorrow and test them out once you get them up. anyways have a good night!

[NicoPlaysThings]

I have tried putting Ravendawn in the exclusion list and it is still crashing on ending soonest mode.

dump.zip

[JourneyOver]

@NicoPlaysThings Hopefully tomorrow we will have this issue sorted out as we may have finally narrowed down possibly what is causing this, until then the only option really is set your priority mode to priority list only.

[DevilXD]

This should now be fixed in the latest master via https://github.com/DevilXD/TwitchDropsMiner/commit/d333c86598ba00871a1b24a3b93d2c5954dd93a2

[JourneyOver]

So far from at least my very small amount of testing so far (only a couple minutes of switching back and fourth between priority modes and reloading things) it no longer seems to be crashing with my main account. So yep I'll say this looks to indeed be fixed now :D