search

soft-networks / is-this-thing-on

is this thing on?
5 stars 0 forks source link

Use Stream instead of Mux for livestreams #36

Closed kyeah closed 3 months ago

kyeah commented 3 months ago

Changes

Context

This is a WIP branch that showcases how we can use Stream instead of Mux for livestreaming. In addition to potential pricing benefits, Stream also appears to provide lower latency via both WebRTC and through the traditional protocol (RTMP via OBS).

I am reusing the room.stream_playback_id field in Firebase to store Stream call IDs. It is possible to stream from OBS as per-usual over RTMP, and I have also added a page on each room to prototype streaming directly from the browser over WebRTC.

Note one benefit of streaming from the browser is that we could overlay the chat, stickers etc. on top to mirror the viewer's perspective, while showing the streamer real-time video (instead of their delayed video.)

Testing

I ran some latency tests below. I have not compared overall video quality and did have some quality drops with the default call settings but could have been my internet.

Mux test (RTMP via OBS + HLS streaming, latency: 7-10 seconds)

https://github.com/user-attachments/assets/79e5c6c1-fb62-4412-a6e5-4263988d1537

Stream test (RTMP via OBS, no HLS, latency: ~500-1000ms) Couldn't get this one to work while also screen-recording for some reason, so took an iphone vid 😂

https://github.com/user-attachments/assets/cb57d23e-c98b-4c32-8e12-5848d7f250a0

Stream test (WebRTC via browser, using webcam, latency: < 300ms)

https://github.com/user-attachments/assets/b1af8f9d-b768-45a1-b6e6-90a420a6114e

Stream test (WebRTC via browser, using OBS Virtual Camera, latency: < 300ms) CleanShot 2024-08-13 at 21 52 06@2x

Instructions for testing

  1. Update client and server env files

    • Add NEXT_PUBLIC_STREAM_API_KEY for the client
    • Add STREAM_API_KEY, STREAM_API_SECRET, and STREAM_ADMIN_USER_ID for the server

  2. Choose a room for testing. Delete stream_playback_id in Firebase and set the room status to active

  3. Visit http://localhost:3000/<roomId>/stream. This will generate a new Stream call and insert it into Firebase as stream_playback_id

  4. Share video/audio access in browser and click Go Live

  5. Open another window and visit http://localhost:3000/<roomId> to view the room.

Alternatively, you can create a livestream in getstream.io manually and write that to Firebase. Can still stream through the browser or send over OBS using the RTMP url.

TODO

There are some additional changes that would be needed for the switch:

Deployment

Relevant Issues

Relates to #21

netlify[bot] commented 3 months ago

Deploy Preview for is-this-thing-on canceled.

Name Link
Latest commit ee601e3248bbc332e09bd1f02a652334e544e750
Latest deploy log https://app.netlify.com/sites/is-this-thing-on/deploys/66c4ad161d13400008fd5d0c
bhaviksingh commented 3 months ago

Yo Yo! Made some changes. This is really looking great.

  1. Had a small fix needed in creating the call, so that you don't have to manually create the call in the stream console.

  2. Set up web hooks! They work really nicely for call started, but I haven't figured out the right web hook for call ended, and it sometimes works, sometimes doesn't. I think we may need some changes on the client to do this - perhaps instead of people going Live, they're joining and leaving the call, and there's a button to override ending the call? Not sure of the inner workings though.

https://github.com/user-attachments/assets/7e8748cb-07fd-4610-8d61-c26aff6c037d

kyeah commented 3 months ago

Nice thank you! re: webhooks — yeah...looks a little wonky.

I think showing an "End Call" button similar to the UI we get in the Stream webapp might be prudent for us. I'm not sure if there's a functional difference between Stop Live and End Call other than whether or not we can re-use the call (e.g. are there price implications to not ending the calls?)

For now I added a custom event that's emitted when clicking "Stop Live", that seems to work reliably and very quickly from my testing.

kyeah commented 3 months ago

A few more notes from testing:

Given these notes I wonder if it's worthwhile to consider supporting both services for any streamers that want to stream 4K and are okay with a few seconds of latency.

kyeah commented 3 months ago

A little messy but I think this is in a good spot now.

  1. Improved the Admin streaming UX to show instructions for either RTMPS or Browser-based (WebRTC) streaming.
  2. Split "Going live" (i.e. starting/stopping the livestream) from "Joining or Leaving" the call, which allows streamers to start/stop the livestream but still use RTMPS/OBS instead of browser-based streaming.
  3. Set up webhooks/firestore so stream_status reflects whether the call is "live". If it is live and nobody is actually streaming video, viewers can still join the room and see the chat/stickers etc. but there will be an "offline" message overlaid.

I tested a few different flows:

RTMPS and browser streaming:

https://github.com/user-attachments/assets/2c2e058a-a2cf-4d8f-bb42-17a1e083d4b4

"offline for now"

CleanShot 2024-08-20 at 10 42 52@2x

kyeah commented 3 months ago

Suggestions for next steps:

Merging code to production

Testing with streamers