Migrate the Viewer List and the /chatters command to Helix API

[Felanbird]

As part of Twitch's announced deprecation of IRC -based commands (I don't remember where the announcement is for this specific one) the Viewer List and subsequently the /chatters command need to be migrated to use the relevant Helix API endpoint.

Helix API reference: https://dev.twitch.tv/docs/api/reference#get-chatters

It's mod only now so we'll need some form of error for /chatters and the viewer list for non-mods.

[cbclemmer]

@Felanbird @pajlada I'd like to try migrating this

[pajlada]

I'd like to try migrating this

Assigned, thank you! Let us know if you run into any issues or need any sort of assistance

[cbclemmer]

@pajlada :tada: Thank you, do you know where the old documentation for https://tmi.twitch.tv/group/user/[username]/chatters is, I can't find it? I'm trying to see if the old version supported pagination and maybe this should too?

[pajlada]

@pajlada tada Thank you, do you know where the old documentation for https://tmi.twitch.tv/group/user/[username]/chatters is, I can't find it? I'm trying to see if the old version supported pagination and maybe this should too?

The old API is undocumented actually, but it does not have pagination, it just outputs all chatters in a single request.

For the Helix API, we'd want to support pagination.

[cbclemmer]

I've been messing with this and it looks like the endpoint being mod only now will hurt some of the functionality like the viewer list and the chatter count. The app can get around this by dynamically creating a list of active chatters for every unique chatter that sends a message. Maybe that needs some sort of expiration time so that really old chatters aren't counted in this list, but the app could even use the messages loaded when loading the chat of a streamer for the first time to get some of the most recent chatters. This would probably alleviate the username autocomplete feature which is probably what is most important to most users. For mods, they would have a better picture of active chatters and chatter count so I think the app should still use the helix endpoint for them. From what I can find, twitch itself has never acknowledged the tmi chatters endpoint, but I would assume they are going to eol it when the rest of the irc commands are depreciated on Feb 18, 2023. The app can use the tmi endpoint as a primary method until it's dead but at least when it dies it would do so gracefully and the app would continue to function. When that happens only a few function calls will need to be removed as well as the tmi endpoint function (that I'll create). As far as the pagination goes, the app would have to capture mouse wheel and scroll position like most infinite lists on the web or use a "more" button, one is obviously easier than the other. The maximum returned chatter list length is 1000 so some kind of pagination has to occur or just be lazily loaded in the background. The /chatters command did not query the tmi endpoint directly but relied on the data already fetched from the tmi endpoint so the new way has to get around that. I'll summarize the proposed features below.

Proposed Features:

• Use api.twitch.tv/helix/chat/chatters when users are moderators of channels. This will allow moderators as much information as twitch deems necessary (who counts for an "active" chatter and that type of thing).
• Dynamically create a chatter list from received messages for use on the viewer list and autocomplete for normal users.
• Periodically cull chatters from active chatter list based on last message received which is how I assume twitch does it on the back end for the helix api.
• /chatters command will be based on the aforementioned features whether the user is a mod or not.
• Keep the tmi chatters endpoint functionality but gracefully fail to the aforementioned features when it finally dies. (probably on Feb 18, 2023)
• Paginate viewer list for mods using a "More" button or based on scroll position in viewer list.

[cbclemmer]

Let me know your thoughts devs

[jupjohn]

@cbclemmer I feel like this faces the same problems as /mods and /vips. I personally think we should only rely on the Helix response and if you're not a mod then a messaging saying "use the Twitch website" should be displayed (as seen in #4053). Implementing these hacky approaches in c2 doesn't match the same functionality provided by Helix and therefore paints an inaccurate picture of the chatter list.

This is on Twitch to open up these endpoints to standard users over Helix like they do for GQL

[cbclemmer]

Implementing these hacky approaches in c2 doesn't match the same functionality provided by Helix and therefore paints an inaccurate picture of the chatter list.

@jammehcow From my testing this is how twitch does it in javascript on the web app. Once the tmi endpoint is gone then autocomplete mentioning people will be broken in chatterino. You can @ anyone in chatterino using the data returned from the tmi endpoint at the moment, but you can't do that in the web app until you receive a message from them. The strange thing is that there is no functional equivalent for the api of the "Community" button on the web app. This shows a filtered view of chatters, mods, and vips. I think they'll either remove the "Community" section on the web app altogether at some point or open up an endpoint on the api for it. It's hard to tell which is going to happen so I can understand waiting...

[LosFarmosCTL]

From my testing this is how twitch does it in javascript on the web app. Once the tmi endpoint is gone then autocomplete mentioning people will be broken in chatterino. You can @ anyone in chatterino using the data returned from the tmi endpoint at the moment, but you can't do that in the web app until you receive a message from them.

That is how twitch fills their autocomplete menu for mentions, but for the viewerlist and the chatters helix endpoint they just have their own data about which users are actually connected to a channel on their IRC/TMI server. Autocomplete in chatterino should definitely switch to completing usernames from message history, but I agree with jammehcow that pulling recent chatters would not be a good implementation for the viewerlist.

The strange thing is that there is no functional equivalent for the api of the "Community" button on the web app. This shows a filtered view of chatters, mods, and vips. I think they'll either remove the "Community" section on the web app altogether at some point or open up an endpoint on the api for it. It's hard to tell which is going to happen so I can understand waiting...

Twitch does not care about 3rd and 1st party API parity, there will be no API endpoint at all for this, 100% guaranteed. The filtered list on the website is there to stay though if I understood everything they said in the last TwitchDev livestream announcing the chatters endpoint correctly, so at least they don’t just restrict 3rd party APIs like they did with /mods and /vips, but even the 1st party API will only work for mods (unless they did the same shit as with their recent-messages 1st party API, which just works with any user token but is only called when you are a moderator. But using GQL is out of the question for chatterino anyways, so for us it doesn’t matter)

• Keep the tmi chatters endpoint functionality but gracefully fail to the aforementioned features when it finally dies. (probably on Feb 18, 2023)

Might be staying around longer, the new chatters endpoint has nothing to do with the commands deprecation directly, so far all they have given us in the mentioned livestream is that they plan to remove the undocumented TMI endpoint eventually, but no details as to when.

[cbclemmer]

@LosFarmosCTL Yeah, I figured the community tab was piped in from the back end somehow but capturing network traffic on twitch hasn't been helpful with telling what is actually going on. I agree with you about the viewer list/autocomplete stuff. I just thought since it would already be built then it would be useful to use in both places, but it's probably too confusing for most users and I may have been reading @jammehcow 's response wrong about that.

[pajlada]

Dynamically create a chatter list from received messages for use on the viewer list and autocomplete for normal users.

Periodically cull chatters from active chatter list based on last message received which is how I assume twitch does it on the back end for the helix api.

We already have one of these in each TwitchChannel object since they inherit from the ChannelChatters class. ChannelChatters keeps an least-recently-used cache of chatters which is currently fed both by incoming chat messages and by the TMI Chatters API. Post-migration, I'm ok with ChannelChatters only being fed by incoming chat messages, and leaving the Helix Get Chatters API only to be used for the "chatter list".

Keep the tmi chatters endpoint functionality but gracefully fail to the aforementioned features when it finally dies. (probably on Feb 18, 2023)

Since there's no 100% deprecation date, I'm not super keen on doing any sort of time-gating of this functionality. Given the amount of extra complexity that would be required to keep using the TMI endpoint until it dies (especially since we don't know how it would die) I'd prefer to just get rid of it entirely in the migration.

Paginate viewer list for mods using a "More" button or based on scroll position in viewer list.

I would prefer not having to press a "More" button or scroll down to load more chatters, I'd rather we just do pagination up to a maximum (e.g. 50 requests) whenever the user clicks the "Chatter List" button and provide everything we got

/chatters command will be based on the aforementioned features whether the user is a mod or not.

This seems fine to me. If you're a moderator: use the total field from a single request of the Helix API. If you're not a moderator: use ChatterSet count.

[LosFarmosCTL]

I would prefer not having to press a "More" button or scroll down to load more chatters, I'd rather we just do pagination up to a maximum (e.g. 50 requests) whenever the user clicks the "Chatter List" button and provide everything we got

Especially for searching the list this would be the better solution. Maybe some sort of loading indicator, but if you have to scroll down to actually load all chatters, search becomes pretty useless.

[cbclemmer]

This all sounds good, I've got it mostly working but I'll probably wait until #3971 is closed so that I can add the mods to the viewer list to create the pr.

[jupjohn]

@cbclemmer I'll be starting on that either tomorrow night or on Saturday as I'm a bit tight on time. Might tag you in the PR to get your thoughts on the pagination (should be simple anyways).

[cbclemmer]

@jammehcow No worries, check out this function for how I did the pagination. Maybe there's a better way send less parameters to the function but I like it being recursive at least.

[jupjohn]

@cbclemmer that was more or less the approach I was going to take, awesome!

[cbclemmer]

New PR is up #4088

[Felanbird]

Re-opening this so #4117 can re-close it 😎