YouTube simulcast support

[mrphlip]

Brainstorming what would be good for the chatbot to have, should LRR decide to go ahead with the youtube simulcast thing...

• Ability to talk to YT's chat API (all I know is it's not IRC, some sort of webservice polling I think)
• Automatically detect when they're streaming on YT, and join the chat (since there's a chat per video, not just a single chat channel like Twitch)
• Read messages, log them for prism and vod playback
• Respond to chat commands
  • Recognising YT members for sub-only commands, and YT mods for mod-only commands
• Log new subscribers, memberships, superchats, for notifications (potentially to bundled in the overlay with Twitch follows, subs and bits, respectively?)
• Log moderator actions to Slack (if possible)
• Automated messages that aren't responses should go to both chats (eg: card viewer)
• Apply spam filters to YT chat?
• Maybe: allow cross-linking of Twitch users and YT users, so that Twitch subs can use sub-only commands in YT chat, and vice-versa... like the linking we already have for Patreon?
• Maybe: allow users to log in to lrrbot.com with their YT account, instead of their Twitch account?
• Does YT chat have some equivalent of "whispers" we want to hook into?

[mrphlip]

Implementation brainstorming:

We already have the ability to bold external message sources into the IRC-client backend... for Twitch whispers, we have some processing to make it so incoming whispers are fed into the backend as private messages, and then any private messages the bot sends are turned into outgoing whispers before they're sent. So we know this is possible.

So we could do something similar to integrate YT... have the bot connect to the YT chat, and feed any messages received into the IRC-client backend as though they were messages in, say, the #youtube channel, and then any outgoing messages to that channel can be intercepted and sent back to the current YT chat?

[andreasots]

My notes going through YT's API documentation:

• Ability to talk to YT's chat API
• Read messages

LiveChatMessages: list and LiveChatMessages: insert

• Automatically detect when they're streaming on YT, and join the chat

There doesn't seem to be a concept of joining. Basically you have to somehow find a chat ID and then start fetching messages.

There's LiveBroadcasts: list with mine=true but it would need the broadcaster's OAuth token. The chat ID is items[].snippet.liveChatId.

Alternative the Discord bot currently does this:

• fetch the channel requesting the contentDetails part: Channels: list
• get the uploads playlist from items[].contentDetails.relatedPlaylists.uploads.
• poll the uploads playlist: PlaylistItems: list
• for each video it hasn't seen before:
  • fetch the video requesting the liveStreamingDetails part: Videos: list
  • check that items[].liveStreamingDetails is present.

From the last step we could get the chat ID from items[].liveStreamingDetails.activeLiveChatId

There's also PubSubHubbub but I'm choosing to ignore it because it's XML.

• Does YT chat have some equivalent of "whispers" we want to hook into?

I don't think so.

• Recognising YT members for sub-only commands, and YT mods for mod-only commands

This is in the authorDetails of the chat message:

• Members: authorDetails.isChatSponsor
• Mod: authorDetails.isChatModerator. I think authorDetails.isChatOwner should be for the broadcaster who should also count as a mod.

• Log new subscribers, memberships, superchats, for notifications

I believe it's impossible to get subscribers in that direction (there is no 'who is subscribed to me' request, only 'who am I subscribed to'). I think the rest is in the chat message. BTW: members used to be called sponsors and the API hasn't been updated to reflect that.

• Log moderator actions to Slack

There is a snippet.userBannedDetails on the chat message. The author of the message is the moderator.

Implementation:

• allow cross-linking of Twitch users and YT users
• allow users to log in to lrrbot.com with their YT account, instead of their Twitch account?

I think this needs a radical new concept of a user without a Twitch account. We should look around for Flask extensions that could do most of the account and auth stuff for us. I found Flask-Dance but maybe there's something even better.

I considered doing it for Patreon but I decided not to do it because we still had votes and I didn't know what to do with those when merging users together when linking.

converting YT chat into IRC

The channel name should contain the chat ID because I suspect there can be multiple active chats at a time, for example between streams the old chat is still active after a stream for a bit but also a new stream has started. Also the channel name schema should be such that if it accidentally gets sent to Twitch then Twitch would reject it. So maybe something like &youtube:LIVE_CHAT_ID.

I don't know what to do with the lack of direct messaging. Although if we do nothing and just discard them then the experience would be pretty much the same as on Twitch where whisper delivery is basically guaranteed to not happen because of default permissions. (By default whispers from strangers are blocked and moderators in a chat where you just advertised your viewbotting service are considered strangers.)

[paul-lrr]

As an update on LRR youtube stuff, a priority would be to get youtube members integrated into LRRbot for sub welcomes. Not only because we want to thank those people, but also because we want to use "get a shout out on the stream" as the required youtube membership perk.

Looking at the youtube API docs, there is an endpoint for listing the current youtube members (https://developers.google.com/youtube/v3/docs/members/list). Unfortunately, there isn't a notification system for new members, but the list endpoint does have an updates mode which looks like it is intended to be polled to find new members. Alternatively, even if we were only able to pull the full list into LRRbot, we could do something where we thank the whole youtube list at the same time, once a month, during LoadingReadyLive or something like that.

[andreasots]

New events that will be added soon™:

"youtube-membership"

A user has become a channel member. Almost the same structure as "twitch-subscription". Counts toward the combined storm count.

Normal version:

{
    "time": str,
    "name": str,
    "channel_id": str, // user's channel ID
    "avatar": str,
    "tier": Optional[str], // membership level name
    "is_upgrade": bool, // whether the user just upgraded from a lower level
    "count": int,
}

Gift version:

{
    "time": str,
    "name": str,
    "channel_id": str, // user's channel ID
    "avatar": str,
    "benefactor": str,
    "tier": Optional[str],
    "ismulti": bool, // whether this event is part of a multi-gift drop
    "count": int,
}

"youtube-membership-milestone"

A user has sent a Membership Milestone Chat. Almost the same structure as "twitch-resubscription". Counts toward the combined storm count.

{
    "time": str,
    "name": str,
    "channel_id": str, // user's channel ID
    "avatar": str,
    "tier": Optional[str],
    "monthcount": int,
    "message": str,
    "count": str,
}

"youtube-membership-gift"

A user has gifted one or more memberships. Almost the same structure as "twitch-subscription-mysterygift". Unlike "twitch-subscription-mysterygift" it gets sent for even a single gift membership.

{
    "time": str,
    "name": str,
    "channel_id": str, // user's channel ID
    "avatar": str,
    "count": int, // NB! this is the number of memberships gifted and not a storm count
    "tier": Optional[str], // membership level name
    "members": List[YoutubeMembership], // list of `"youtube-membership"` payloads (the gift version) without `"time"`
}

"youtube-super-chat"

A user has sent a Super Chat.

{
    "time": str,
    "name": str, // user's channel name
    "channel_id": str, // user's channel ID    
    "avatar": str, // user's avatar
    "amount": str, // amount paid (e.g "$1.00" for a 1 USD Super Chat)
    "amount_micros": int, // amount paid in micros of the purchase currency (e.g 1000000 for a 1 USD Super Chat)
    "amount_currency": str, // ISO 4217 code for the purchase currency (e.g USD)
    "level": int, // Super Chat tier based on the money spent
    "message": str, // user's message
    "count": int, // storm count
}

"youtube-super-sticker"

A user has sent a Super Sticker.

{
    "time": str,
    "name": str, // user's channel name
    "channel_id": str, // user's channel ID
    "avatar": str, // user's avatar
    "amount": str, // amount paid
    "amount_micros": int, // amount paid in micros of the purchase currency
    "amount_currency": str // ISO 4217 code for the purchase currency
    "level": str, // Super Sticker tier based on the money spent
    "sticker_id": str, // ID of the sticker
    "sticker_url": Optional[str], // URL for the sticker image
    "alt_text": str, // the sticker's alt text
    "alt_text_language": str, // the BCP 47 tag for the language the alt text is in
    "count": // storm count
}

EDIT: added "sticker_url" to youtube-super-sticker.

[andreasots]

I've merged and deployed the initial version of LRRbot's YouTube integration.

Changes:

• Users and accounts refactor: all remote (OAuth) accounts live in the accounts table and the users table is used to tie multiple accounts together.
  • Since I expect users to pop into and out of existence as accounts get linked and unlinked I've tied the auditing columns (what used to be history.changeuser and clips.rater) to accounts and not users. Those columns get set to the account you logged in with.
  • The next step is to add an account linking/unlinking page and add all account providers to the login page.
  • Also #435 should now be possible to implement.
• Add YouTube as an account provider.
  • Logging in is not implemented but enough is there so that we can get the two OAuth2 tokens we need.
  • We need to get LRRbot verified (all the scopes we need are 'sensitive' and we'd probably want to let more than 100 users log in) so until we're ready to do that LRRbot will be set to be 'in testing' and we need to whitelist the accounts we need.
• Implement reading the YouTube chat
  • Notifications get created for new members, membership milestones, gifts, Super Chats, and Super Stickers.
    • The new events are described here: https://github.com/mrphlip/lrrbot/issues/1503#issuecomment-1942415127
  • Mod actions get logged for deleted messages, banned users, and members-only mode being activated or deactivated.
    • Also YouTube might not even send deleted messages and banned users.
  • We don't have enough query quota to read the chat at a reasonable rate. We have a daily quota of 10,000 units, every LiveChatMessages: list is 5 units and it recommends us to poll it about once a second so we run out of quota in about half an hour.
    • The solution is, of course, more bureaucracy: we need to get audited and then we could request more quota.
    • The current workaround is that we only poll at most once a minute across all chats.
  • Implement the new notifications.

Roadmap:

1. Create notifications for members who joined outside a stream. LRR needs to contact their YouTube rep to get us access to Members: list.
2. Implement chat commands.
3. Implement spam filters.
4. Implement the chat log.

[paul-lrr]

wow, this looks great! It is so nice to have LRRbot integrating and normalizing the notifications from the different systems instead of having to build all that functionality into each overlay or widget that I make. Do you need anything from me to put this into production? Not sure if LRRbot needs to be given special permissions on the LRR Youtube account or something. Also, will these Youtube membership start showing up on the LRRbot.com/notifications page?

[andreasots]

I've added a fallback for getting active streams without LRR's token and enabled joining YouTube chats in LRRbot.

Also, will these Youtube membership start showing up on the LRRbot.com/notifications page?

Yes.

[paul-lrr]

A couple of questions: With the above update, does that mean that you don't need an LRR login token to get new/recurring memberships, superchat, etc? Does this get the info when LRR isn't streaming to youtube?

[andreasots]

With the above update, does that mean that you don't need an LRR login token to get new/recurring memberships, superchat, etc?

The only thing that the fallback changes is how LRRbot discovers streams. With the fallback LRRbot will join the chat when the stream goes public. With LRR's token LRRbot would join the chat when the stream is created.

Or rather up to five minutes later because LRRbot checks streams every five minutes.

Does this get the info when LRR isn't streaming to youtube?

No. Fetching members outside of a stream with Members: list requires LRR's token and also LRR to contact their YouTube rep to give us access.