search

GRVYDEV / Project-Lightspeed

A self contained OBS -> FTL -> WebRTC live streaming server. Comprised of 3 parts once configured anyone can achieve sub-second OBS to the browser livestreaming
MIT License
3.64k stars 137 forks source link

Feature: 10-bit HEVC + OPUS streaming #15

Open redzic opened 3 years ago

redzic commented 3 years ago

Right now, it's possible to stream (to for example youtube or twitch) by encoding a video in advance and steaming with ffmpeg -re -i input.mkv -c copy -f flv rtmp://{url}. However, right now, twitch (for example) only supports 8-bit h.264 and AAC. Youtube, in a sort of unofficial way, also allows streaming with HEVC (I think with HLS) but it transcodes the video so the quality is usually worse than h.264 on twitch anyways.

For the best possible quality, it would be amazing if lightspeed could support 10-bit (or even 8-bit would be an upgrade) HEVC streaming with the OPUS audio codec. Opus is, right now, the best lossy audio codec, and can reach transparency at pretty low bitrates (much lower than AAC can anyway, even with the apple AAC encoder, not to mention opusenc also generally encodes faster than the AAC encoders out there).

GRVYDEV commented 3 years ago

Hi! As of right now we are using H264 and OPUS. I will need to look into HEVC. I am not super familiar with the different video codecs etc. but this is something I definitely want to learn more about!

redzic commented 3 years ago

Oh wow! That's pretty amazing, I didn't know of any other live streaming website that supports streaming in Opus. But yeah, I stream some stuff on twitch (mostly just to watch stuff with my friends) but there's always a sort of hard cap on the video quality since 8-bit h264 just isn't that good in quality (especially when it's encoded in real time). It would be really nice if the same visual quality could be achieved with less bitrate. If there's anything I could do to help get HEVC get implemented let me know, as I do have some rust experience.

If HEVC ends up being a little out of scope for the project, then if possible I think at least 10-bit h264 would still be a good step up in quality from just 8-bit h264.

redzic commented 3 years ago

And ofc, good luck with the project in general and I understand whichever direction the project decides to take.

GRVYDEV commented 3 years ago

So as of right now it appears that HEVC is not supported by the RTP transport. The valid MIME types can be seen here https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-2 We are limited by a bunch of different things in terms of what kind of video we can carry. For example, we are limited by what OBS will send as well as what RTP can carry since that is the underlying video transport protocol

GRVYDEV commented 3 years ago

So as of right now it appears that HEVC is not supported by the RTP transport. The valid MIME types can be seen here https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-2 We are limited by a bunch of different things in terms of what kind of video we can carry. For example, we are limited by what OBS will send as well as what RTP can carry since that is the underlying video transport protocol

I may be misinformed here. Upon reading the rtp-parameters I noticed that opus is not there however we are using opus so maybe we could transport HEVC? I will need to look into this more

redzic commented 3 years ago

That's what I was checking for as well, I searched AAC and it's not listed there either.

I found some information about it:

RTMP protocol supports H264 but does not support HEVC because it's underlying container FLV does not support HEVC. So unless Adobe specifies it in the specification, it is unlikely to be supported by anyone. The spec was defined quite a while ago when most modern codec was H264.

(source)

That's what I'm wondering as well. It must be that you're not using FLV at all, since I just checked (just from using ffmpeg) and it's saying that neither HEVC nor OPUS are supported in an flv container. I guess what matters is just the underlying container.

Also, I think OBS should be able to send an H265 stream just fine, since I think it just uses ffmpeg underneath and from my own testing ffmpeg can send any stream just fine (or at least, it can send HEVC to youtube via HLS, although youtube doesn't really seem to have super strong support for it, because the only place it's mentioned is in some developer doc rather than a like normal user-facing guide or anything).

GRVYDEV commented 3 years ago

That's what I was checking for as well, I searched AAC and it's not listed there either.

I found some information about it:

RTMP protocol supports H264 but does not support HEVC because it's underlying container FLV does not support HEVC. So unless Adobe specifies it in the specification, it is unlikely to be supported by anyone. The spec was defined quite a while ago when most modern codec was H264.

(source)

That's what I'm wondering as well. It must be that you're not using FLV at all, since I just checked (just from using ffmpeg) and it's saying that neither HEVC nor OPUS are supported in an flv container. I guess what matters is just the underlying container.

Also, I think OBS should be able to send an H265 stream just fine, since it probably just uses ffmpeg underneath and from my own testing ffmpeg can send any stream just fine.

I really doubt RTMP will ever support HEVC since I dont think it is being maintained anymore. Afaik RTP does not use an underlying container in the sense that RTP is the container. If I am correct that means I can send whatever I want through RTP which means we could support HEVC. You could do some testing by starting up the ingest and telling OBS to use HVEC and then grab a pcap of the incoming packets to see if OBS will send HEVC RTP packets. The video packets are type 96 so look for those in the pcap

Edit: Confirmed that RTP can carry whatever we want it to. Then only thing we need to do is wait for browser support of HEVC

GRVYDEV commented 3 years ago

Here is another thing to think about though... Browser support https://caniuse.com/hevc looks like safari is the only one that supports HVEC at the moment

redzic commented 3 years ago

Hm, from what I'm reading it seems like browsers offload the decoding to the underlying OS so in a sense it might still work, I think it'll have to be tested out. I think it has to due with HEVC licensing being really expensive and browsers not wanting to pay those licensing fees just to implement a video decoder.

Another option for a more modern video codec is AV1 (which is actually even better than HEVC in terms of efficiency). Right now, we're only just getting hardware decoders (on some android phones as well as the newest generation of AMD + NVIDIA gpus, and also various other devices like TVs even) and such, but libdav1d (the fastest AV1 decoder) is actually pretty fast, especially for 8-bit content (and eventually they'll have optimized AVX2 for 10-bit as well). Anyways, as far as I'm aware, YouTube uses AV1 for basically most videos with a lot of views so it's not like an unheard of codec or anything. While it's probably not as practical as HEVC for real time encoding (live streaming) (at least right now, but encoders are always getting faster), it still can fill a somewhat niche use case that I have where I can encode videos ahead of time and stream them via ffmpeg.

Baa14453 commented 3 years ago

Browser support for HEVC is low due to patent nightmare: https://bugzilla.mozilla.org/show_bug.cgi?id=1332136 But maybe worth looking into AV1 for future? https://caniuse.com/av1 Remember transcoding in real-time will put a CPU burden on your server

redzic commented 3 years ago

Yeah, I think AV1 support would be pretty awesome, even though it might not be so practical for most people (at least for real time encoding, decoding support is actually quite good like I said earlier) at the moment. Still, if it could be at least implemented at the protocol level that would still be very good.

krakow10 commented 3 years ago

The SVT-VP9 encoder is quite quick and can be added to ffmpeg with a patch available in the repository. VP9 is comparable to HEVC in video encoding efficiency and is widely adopted in browsers because of YouTube.

GRVYDEV commented 3 years ago

The SVT-VP9 encoder is quite quick and can be added to ffmpeg with a patch available in the repository. VP9 is comparable to HEVC in video encoding efficiency and is widely adopted in browsers because of YouTube.

Its looking like OBS may remove FTL support meaning a separate fork of OBS will be created to maintain it also meaning that we will have more control over what we can do at the protocol level so thats something to think about in the future.

krakow10 commented 3 years ago

Perhaps ffmpeg could be built to include FTL streaming, and then use custom ffmpeg output on OBS. Perhaps a patch like that already exists somewhere. That would also make it much easier to stream in non-H.264 and with multiple audio channels.

Artoria2e5 commented 3 years ago

Chinese streaming companies and CDNs have encountered the same issue, and what they ended up doing was extending the FLV format contained within RTMP to carry H.265. An example spec is available at https://github.com/ksvc/FFmpeg/wiki. Some also allegedly have assignments for VP9, Opus, and AV1, but I am unable to locate the definitions for now. (Tweet linked is from author of flv.js.)

I think there is something to be done if Lightspeed can get some orgs to come together and agree on some extensions. (Documentation and cooperation... not exactly a strong suit for our corps. Sorry.) It's all crazy and magical, but hey FLV is magical in its choice to not use FourCC and just give 4 bits. The best way out might be to use a value "15" and put a FourCC in the frame instead.

Anyway, this is a trivia. I don't really think there is a lot of a point in beating a dead horse when you can clearly use non-FLV stuff.