Roadmap for 2023

[getroot]

Happy new year, everyone.

As many of you know because I've mentioned it several times, we spend the first half of the year working on OvenMediaEngine and the second half doing external development projects for survival. I sincerely ask you to become a Github Sponsor so that OvenMediaEngine project can be boosted. Thanks to your contributions, OvenMediaEngine continues to evolve.

Here are our big plans for 2023.

IPv6 support This is far too late. We will unconditionally deal with this within the first quarter of 2023.

Stabilizing the WebRTC Provider Now we can spend some time on this. We will fix everything so that WebRTC input works reliably (especially with OvenLiveKit).

Improving Performance and Stabilizing GPU Encoder In the meantime, the community has been asking a lot about this. We will look into this closely.

WHIP support We will support WHIP once the WebRTC Provider is stabilized. The time has come to support this.

Stabilizing the SRT Provider A number of issues have also been raised about the SRT input. Now we can see this.

Advanced AdmissionWebhooks I found that there are many cases in which the callback of AdmissionWebhooks is linked to their statistics system. We will add meta, such as passing user-agent information, and solve problems such as disconnect events not occurring.

Check and block input quality If a stream of several tens of megabits or a stream containing b-frames is input, OME may not operate normally. We will review the ability to limit this. This work is a low priority and the plan may be delayed if it is determined that there are too many side effects after design.

Fallback stream We will develop a function that can send a video file that can be replaced when the input of the main stream is not live.

Origin Pulling Query String We will develop a function where Edge creates a stream by reading the origin url from the playback URL's query string and pulling the stream from the origin. For example ws://edge.com/app/stream?originUrl="ovt://origin.com/app/stream" would work.

RTMP Pull Provider We will develop RTMP Pull Provider function. This will make it easy to serve WebRTC or LLHLS streams using an existing RTMP Server. If used together with the Origin Pulling Query String function above, OvenMediaEngine will be valuable as a proxy server that relays the existing RTMP server.

Encoding Profiles Encoding profile will be provided so that you can set more diversely.

Various API functions Several APIs are planned, such as certificate reload API and stream deletion API.

Besides this, we have many small features, bug fixes and performance improvements planned. Which feature are you most looking forward to? Or is there something you think is a must-have feature that isn't listed here? Please feel free to comment as much as possible.

[naanlizard]

Re: blocking streams with too high a bitrate or b-frames, perhaps three levels.

1. No checking of input streams

2. Streams checked but not blocked - generate callback or set flag you can query from API

3. Streams blocked

or more practically, just add an API endpoint to query for various stream stats. This may already exist for some things (bitrate?) but I don’t believe exists for b frames

Another thought on the b frame detection, does that take time from the stream beginning? The ux is maybe bad for users if they connect successfully and they are disconnected after a minute or whatever because of b frames being detected. Unless we can send back a custom message to explain the disconnect? I’m not sure if that is possible

Personally we would be interested in knowing if streams are using b-frames, so we can offer or not offer the webrtc player as appropriate

[getroot]

@naanlizard Thank you for your comments. We will add various information to the stream information API, including whether b-frames are included or not. And if you use the stream deletion API that will be added, you will be able to block streams on your own.

[naanlizard]

Is the stream delete api just a forcible disconnect from the server side? Disconnect might be the better term if so. Delete makes me think they can’t reconnect 🤔

[command-tab]

Maybe it is (or, will be) effectively a server-side disconnect but necessarily transmitted as an HTTP DELETE call?

[getroot]

It is correct to forcibly close the connection on the server side. But that's the result of deleting the stream.

Actually I will add API like below (currently only GET <Stream> is provided).

DELETE https://host/v1/vhosts/vhost/apps/app/streams/stream

This API does not care about reconnecting terminated sessions. If the user wants to prevent a disconnected session from reconnecting after an API call, AdmissionWebhooks should be used.

To forcibly disconnect the session from the server(To use the term Disconnect session), the following API should be provided. (The stream will be deleted as a result.)

POST https://host/v1/vhosts/vhost/apps/app/streams/stream:banSession
{
   sessionId: "f9dafjda9121"
}

or

DELETE https://host/v1/vhosts/vhost/apps/app/streams/stream/sessions/<session id>

And this can cause misunderstanding that this function can be used not only in the session (encoder) connected to the provider, but also in the session (player) connected to the publisher.

[nums]

First, a big thank you for the evolution of your product. The addition of the DVR and the iD3 tags injections will allow us to deploy ovenmedia for several of our customers :)

A feature that would be quite important for us is the ability to create a multi-language stream. Technically we currently do this by sending several audio streams in SRT and we generate an HLS with several audio tracks with FFMPEG. Some of my clients could fund this feature, maybe we can discuss it by email?

Regarding WebRTC, WHIP support would be perfect, is simulcast support planned? I know that there is ABR for webRTC, but I have a slight latency on "degraded" streams and suddenly a very slight desynchronization between the video and the audio, the simulcast in input might be a way of solve this problem? and last thing, is the interpretation of PLI requests under consideration on your side?

[basisbit]

This list reads like my xmas wish list. Looking forward to all the good work you all are doing there! 🎉 🥳 ❤️

[mpisat]

If WHIP gets added, perhaps WHEP can be added too? This way OME can support industry standards. Cloudflare started WebRTC beta with WHIP/WHEP recently and it is going to help spread this standard. Then we won't be locked down to specific players I guess?

[getroot]

@mpisat If there is a player that actually supports WHEP, and my time and stamina allow me to do it, I will work on it.

[mpisat]

According to cloudflare https://developers.cloudflare.com/stream/webrtc-beta/

this one does (TypeScript), there is even a demo page https://www.npmjs.com/package/@eyevinn/webrtc-player

[getroot]

@nums If you would like to add features faster or add your own features for a fee, please send your requirements and your budget to support@airensoft.com.

I need to think more about supporting simulcast in 2023. I'll think more about what the OME benefits versus the effort put in, and I'll tell you later. If OME's ABR's A/V sync doesn't work, that's another issue. If you want to fix it, please post it as a new issue later.

I'm not sure what you mean by "the interpretation of PLI requests". Please elaborate.

[getroot]

@mpisat Thanks for sharing good information. It is of course good news that Cloudflare supports it. Although OvenMediaEngine and Cloudflare's WebRTC service cannot be used together. I decided to support WHIP because OBS started supporting WebRTC and WHIP. Like this, if major players like EXOPlayer, videojs, and vlc support WebRTC and WHEP, I'll be on fire.

[llspalex]

Thank you for your continued hard work. WHIP support is particularly exciting! Are there any plans to add support for codecs such as AV1 or VP9?

[zapf2000]

AV1 support would be great. Open source codec, very powerful, safes a lot of bandwidth and hardware decoding is already supported on various GPU's. Even hardware encoding is now available on new GPU's. So why not implement that.

[nums]

I will re-check WebRTC ABR AV/Sync with the last version of OME and I let you know in a new issue if the problem still occur.

PLI (Picture Loss Indication) is really not a must have but it might be worth thinking about. When a viewer has significant packet loss it must wait for the next keyframe: On a video stream at 2 keyframes per second, we can have 1.9 seconds of freeze if on the client side we have lost too many packets. On the other hand, if we process the PLI request on the server side, we could generate a keyframe "on the fly" and have as little freeze as possible. Afterwards, you have to effectively limit the number of keyframes generated because this can have a strong impact on the quality of the stream.

[getroot]

@nums Ah... you meant RTCP PLI. The WebRTC Provider already requests the PLI to adjust the keyframe interval. But on the Publisher side it's impossible to do that. It will be easier to understand if you assume that tens of thousands of viewers are concurrently connected. This will make every frame a keyframe. Also, streams that bypass Transcoder cannot handle this either. In preparation for such cases, we already provide WebRTC FEC and RTX functions to reduce packet loss, and provide TCP Transport functions. Sending WebRTC over TCP has no packet loss issues, so I recommend you use this.

[basisbit]

WebRTC over TCP just has different packet loss issues. Typically when a peering/transit point is congested and when the distance between edge server and client are big. Additionally, WebRTC over TCP introduces playback delay issues when the playback buffer runs empty. WebRTC over UDP with RED (only on-demand per connection) and RTX and FEC usually is better suited for very low latency playback from my experience. TCP is good if you use medium or big playback buffers and/or if your edge servers are very close (latency) to the users.

[nums]

Thanks @getroot and @basisbit for your feedbacks.

Actually recovering the RTCP PLI request is earlier for videoconferencing use with few participants. I know Mediasoup has for example added a Max Request Keyframe Interval parameter but yes, I don't think it's very interesting to implement it on OvenMedia.

Otherwise, I re-tested with the latest WebRTC ABR version and I actually have no more desynchronization between the different bitrates. In my opinion WebRTC simulcast is therefore not very relevant in view of the very good working of WebRTC ABR.

Thanks for all, the Roadmap 2023 seems perfect to me. I also sent an email to support regarding the management of multilingual.

[stevespaw]

I would love to have SRT-"output" for an SRT Re-stream. This would simplify our workflow greatly!

[kakaruta]

Is AV1 support from the 2022 roadmap missing from the 2023 roadmap?

Does OME have plans to support AV1 Codec?

[getroot]

@kakaruta AV1 will of course be implemented in OME. However, it is not yet widely used in a commercial environment, so it has been pushed out of priority.