RTMP Ingest compatibility

jailuthra commented 4 years ago

Abstract

LPMS currently uses joy4 for RTMP ingest, which fails to work with some RTMP streams (like Wirecast). This is a proposal with various options to improve our RTMP compatibility and overall ingest workflow.

Additional Context & Motivation

While the joy4 library provides a lot of functionality (HLS server, transcoding, transmuxing) we only use it for RTMP ingest, and have our own workflow for segmenting the RTMP stream and of course transcoding it. The only thing joy4 currently solves for us is running a local RTMP server on port 1935 - where the end-user can send a stream to and lpms will listen on it, segment it, transcode it and output the bitrate-ladder HLS stream.

RTMP ingest is a solved problem and most issues we face are due to joy4 being untested in a production environment with non-ffmpeg codecs and mediaservers.

Possible Solutions

1. Fix/Modify joy4

Livepeer already maintains a fork of joy4 with minor fixes for issues faced with Wowza etc. We could continue doing this and fix the existing issues in our joy4 fork.

Pros: Least invasive to the existing workflow.

Cons:

High maintaince cost. A lot of effort has already been spent to get our RTMP ingest to simply work.
No guarantee that our fix for Wirecast will make it compatible with the next weird encoder/media server.

TODO: Test darkdragon's branch with some fixes for Wowza, to see if it works with Wirecast.

2. Integrate with nginx-rtmp-module

There is an open source rtmp-module plugin for Nginx which is quite configurable and works with almost every RTMP stream out there. Instead of using the joy4 go module, we can compile and install nginx with the rtmp plugin and run it with a very simple config to listen on port 1935:

rtmp {
    server {
        listen 1935;
        application live {
            live on;
        }
    }
}

Pros:

Compatible with almost every popular mediaserver/encoder out there.
Easy to configure without writing lot of boilerplate code.

Cons:

Need to compile another external depdency.
Independently execute it at runtime, or link and integrate it within our Go node.

3. Decouple RTMP ingest from the node/lpms

Currently LPMS does the job of a full-fledged media server (by integrating RTMP ingest), but most of the users that run a B-node don't really use the inbuilt RTMP server other than for local testing. Production UGC platforms that use Livepeer usually prefer to use their own RTMP-ingest solution or media server which would do the segmentation for them, and integrate with Livepeer network only for transcoding over HLS.

With the WIP on a pull-mode cli tool or a proxy-B node, it makes more sense to have Livepeer focus only on segmenting and transcoding, and let the user integrate with the mediaserver of their choice.

Pros:

No maintaince cost for us on being compatible with differing client implementions of the RTMP protocol.
The user gets the ultimate choice of integrating with the RTMP ingest that suits them, and don't need to run joy4 in the background by default.

Cons:

The inbuilt RTMP server is useful for local testing.
Somebody's workflow might break and would need to switch.

Proposed Solution

IMO a mix of option 2 (using nginx-rtmp) and option 3 (decouple node from rtmp) would be the best solution.

To not break somebody's existing workflow and to ease local testing, we can document (or even provide a small script for) setting up nginx-rtmp and provide configs that work with the most common usecases. We also leave the segmenting code in place, and ready to be used in case somebody setup a custom RTMP ingest server anywhere in their local network and supplied the address & port while starting the B-node.

But we move away from the model of go-livepeer B-node being a full media server and focus on improving the segmenter & transcoder and making it easy to integrate with something like Mist in future. This also make the livepeer node closer to unix philosophy of doing one job right. Nginx very easily integrates with ffmpeg because of this and thus a livepeer cli tool would fit right in its place.

Alternative solutions

4. Use joy5

From the joy4 readme it seems like the author has focused effort on a new project joy5, which again does a lot of things along with RTMP ingest. We could look into upgrading to joy5, but it doesn't seem as popular and has negligible documentation. The RTMP ingest code again seems to be custom rolled with hardcoded buffer sizes. If we're moving from joy4 anyway then it makes sense to pick something that we are sure will support everything under the sun and is easily configurable i.e. nginx :)

Implementation Tasks and Considerations

TBD.

Testing Tasks and Considerations

TBD.

Known Unknowns

TBD.

kyriediculous commented 4 years ago

Thanks for this, sounds quite clear even for me (albeit that i just skip some of the video workflow specific details).

but most of the users that run a B-node don't really use the inbuilt RTMP server other than for local testing.

Wouldn't this negatively impact users running standalone broadcaster nodes in small scale set-ups that don't have the capacity to provision their own media server ?

jailuthra commented 4 years ago

Wouldn't this negatively impact users running standalone broadcaster nodes in small scale set-ups that don't have the capacity to provision their own media server ?

Ah yes this point came up on discord as well, my bad this wasn't clear from the spec. The plan is to use nginx-rtmp in place of joy4, but it is not enforced upon the user to run the node with RTMP ingest.

We can have a script to download, install, run and integrate the RTMP ingest server with the livepeer node similar to the script that installs ffmpeg and other external dependencies in ~/compiled. This particular script can be optionally enabled/disabled using a docker flag or something similar.

It is up for debate what the default should be for the prebuilt binaries (or maybe we can build both for a release)? But in the end nobody's existing workflow should be affected more than enabling an extra flag while compiling or an extra argument when running the node.

dob commented 4 years ago

+1 for this as long as someone running the node only has to add an extra flag to enable broadcast mode.

darkdarkdragon commented 4 years ago

nginx-rtmp-module seems to be unmaintained - last commit made in 2017, it has 906 open issues and 79 open PRs. That means that with this module we will be in exactly same position as with joy4 - we will be maintaining own fork with own fixes - but it will be much harder because code base much bigger, it written in C, and also it integrates with nginx - which is another big piece of code. Given that all joy4 is doing as "media server" - is just forwarding network packets to ffmpeg - I think for us it will be much easier to maintain joy4.

Other option would be to fully remove our RTMP support (including segmenting code) and just offer open source version of Mist to those who want RTMP. Pros:

Mist is supported and evolving
We already working on integration between Mist and Livepeer Cons:
- Mist still not open sourced their plugin for Livepeer integration

ericxtang commented 4 years ago

I support the proposal. Having seen the pain of maintaining an RTMP library, I can see the benefit of using a third version.

For the default setup, I'd prefer a more Switzerland option. We already have Mist support, and are already in the process of working with them to open source it. In the mean time, I think having something like Nginx or SRS can be helpful to Livepeer's open source users, especially if they already use those software (nginx has 9.9k stars, and srs has 9.4k stars).

darkdarkdragon commented 4 years ago

@ericxtang to clarify - why we bringing this up now? Do we really experience any pain with what we have right now?

ericxtang commented 4 years ago

@darkdarkdragon we aren't experiencing any pain with Mist now. However, as the usage of the public network ramps up, we want to be able to offer a good solution for users who would like to use Livepeer directly. I see this become an issue in a few months. We probably won't work on this right away.

yondonfu commented 4 years ago

I like the proposed solution of a mix of options 3 + 4 such that:

A go-livepeer user can have access to a fully integrated solution with RTMP ingest offered by a open source component such as Mist, nginx-rtmp, etc. that "just works" right out of the box
A go-livepeer user can optionally choose to use a different open source component that offers RTMP ingest and that can integrate with go-livepeer

For clarification, is the expected architecture for an integration with an open source component that offers RTMP ingest based on pulling from the RTMP server (either by pulling using RTMP or by pulling from a watch folder if the RTMP server can write files to a folder)?

jailuthra commented 4 years ago

is the expected architecture for an integration with an open source component that offers RTMP ingest based on pulling from the RTMP server (either by pulling using RTMP or by pulling from a watch folder if the RTMP server can write files to a folder)

Yes the segmenter will pull directly from the RTMP server using ffmpeg's RTMP implementation and convert it to HLS segments. The Cgo lpms_rtmp2hls function which is triggered by the segmenter sheds some light on the flow here.

Thus the current codebase can be modified easily to be used with some other RTMP server implementation, as long as it is on the same address/port that it expects.

livepeer / lpms