Twitch Chat 3D Audience

[smsithlord]

What would this gadget do?

Replace 2D Twitch chat windows that many streamers pin in their VR overlays with a miniature 3D bleacher that shows Twitch chatters as simple 3D avatars sitting in the bleacher. What they say can appear in comic chat bubbles above their little avatar. As users join & leave the chat, it is physically represented on the bleacher. Viewers could even use special chat bot commands to alter their little 3D avatar's color or style.

The goal is to replace the 2D visual representation of a streamer's Twitch chat with a 3D physical representation.

Who would use this gadget?

Small audience Twitch streamers who would normally have their boring 2D chat window pinned to their VR overlays.

Assuming that you're on the team, what other skillsets would you need to make this project happen over a couple days of hacking?

This involves writing a JavaScript Twitch chat client to access the chat & join/exit messages, making 3D models of the bleacher & little 3D avatars for chatters, and creating a method that takes what a Twitch chatter says and writes it to a canvas to position over their avatar's head. I can do all of these things over the course of a couple days, but will gladly split up the work if anybody wishes to assist.

What will be the toughest part of building this gadget?

First, for a JavaScript Twitch chat client to work, the user must type in their Twitch username and paste in their OAuth token into the gadget somehow. I am unsure if this will actually be a challenge to accomplish in Aardvark, but it seems like it very well could be depending on if Aardvark has a way for users to type or paste text into gadgets yet.

Second, making it elegantly scale between drastically differently sized Twitch communities. Large streamers have thousands of viewers and their chat moves insanely fast, so it would not be too useful for them.

To avoid the need to scale, we could simply target smaller Twitch communities that only have < 20 viewers/chatters. This small size actually accounts for the majority of Twitch streamers, including every single one of my friends who Twitch streams. :D

[JoeLudwig]

I'll give some thought to how we could get enough typing in there to let people enter their passwords into twitch. As you suggested in slack, even copy/paste would do the trick there, and would definitely be easier than a full keyboard.

[smsithlord]

After digging through the documentation, it looks like bootstrapping & getting a react scene going will be the biggest challenge for me personally. React does not look similar to making Three.js WebXR apps (which I am familiar with), so learning how to interact with the 3D scene without the Three.js quaternions & vectors I'm used to will be a learning experience for me.

The client-side JavasScript for the Twitch chat, creating the GLBs for the bleachers & chatter avatars, and writing chat text to a canvas (if needed) are things I have experience with that will carry over well though, it looks like.

[JoeLudwig]

I can certainly help get the react scene up and running. The skeleton you get out of npm init "@aardvarkxr" is most of the way there for that. From there drawing some models is trivial.

Another twitch-related thing that would be cool would be to have the streamer's own gadgets (or at least the multi-user ones) be sent out to anyone who is watching them via another stream viewer gadget. The streamer could draw on white boards, or use any of the multi-user gadgets that come out of the hackathon with their viewers. And the viewers wouldn't necessarily need to be logged in, so it might be easier.

One super-lame form of copy-paste: Open the Chrome dev console to the gadget and run some Javascript with your credentials. Enough to get going with development. Maybe not enough to let anyone else use. :)