search

livepeer / go-livepeer

Official Go implementation of the Livepeer protocol
http://livepeer.org
MIT License
546 stars 171 forks source link

Multiple Streams per Broadcaster #481

Closed j0sh closed 5 years ago

j0sh commented 6 years ago

Is your feature request related to a problem? Please describe.

With delayed broadcasting, we had to limit the number of streams to 1 per broadcaster. With a separate job creation step, there is the potential to have many outstanding jobs. What is missing is a way to select between jobs when connecting to a broadcaster. Another nice feature is a 'view-only' testing feature, eg a broadcast that doesn't get sent to a transcoder but is still visible through the relay network.

Describe the solution you'd like

When a job is created, a JobID is assigned. Use this to look up the job. The RTMP connection string becomes something like:

rtmp://host/stream/<jobID>

If jobID does not exist, or the broadcaster is not authorized for the job, it becomes a new, view-only stream with a new StreamID. A job is not created automatically.

We can add an -autoJob flag to the CLI. In this case, if jobID cannot be used, then the behavior will match what we have now: a new job is created automatically, along with a new StreamID, a new JobID, etc. Once the job is created, the user can continue broadcasting with the original connection. If the connection breaks, however, they have to reconnect with the new JobID.

The reason for adding an -autoJob flag is to reduce the possibility of a CSRF attack surreptitiously creating jobs for you on the blockchain. This flag should be disabled by default, but can be very useful for testing.

To preserve the current behavior as much as possible (given that jobs are somewhat short-lived, although we could easily extend them), we could also have a special RTMP path, say /stream/latest , which would pick up the most recent job on the node. Along with -autoJob creating a job if there is no valid stream available, this should match exactly the current behavior of delayed broadcasting. This would help long-running channels such as livepeer.tv without imposing manual intervention in creating jobs.

Implementation steps:

Describe alternatives you've considered

We can use the StreamID rather than JobID, which might be a bit nicer because the StreamID can be created immediately, while we have to wait for the blockchain to confirm the JobID.

However, the StreamID is currently very long, and node IDs have very similar prefixes, making it difficult to visually distinguish one StreamID from another. Moreover, jobs on the blockchain are queried by JobID, not StreamID, which makes the lookup process a bit more involved, especially if the broadcast node was not the one creating the job (so there would be no prior local record of the job).

The JobID or /latest feature is not safe for multi-tenant environments such as a broadcast service. Such a service would need to be careful about authenticating incoming connections to prevent job hijacking. (StreamIDs alone wouldn't help either, since that is stored in the blockchain.) One solution is to generate an unique alias mapping a given stream name to a JobID, but that entails more bookkeeping, especially if the broadcast node is not the one creating the job.

dob commented 6 years ago

Looks great. What's the current expiration policy for a job?

j0sh commented 6 years ago

Right now jobs expire after 5760 blocks, or 1 day. It's a hard-coded value in goclient.

With the v2 network, there is no cost to having a really long job, so we might want to up this limit if users are going to be creating jobs as a separate step. Expiration could also be configurable at job creation time.

j0sh commented 6 years ago

Added a note at the end about this design being suboptimal for shared/multi-tenant environments.

ericxtang commented 6 years ago

This proposal is great. It makes the transcoding network more useful. IMO, this feature in combination with delayed broadcast changes the meaning of a "job". It's now more like a "time-capped transcoding channel".

The "product implication" is interesting. Will we ask the users of the node to figure out how to do this? It's a strange concept for someone not familiar with the Livepeer protocol. Something to consider in the API design.

j0sh commented 6 years ago

Yeah, it's a pretty significant workflow change. The one saving grace is that users already have to make a deposit and set their broadcast preferences (profiles, segment price, etc). In fact, we could probably merge the "broadcast preference" option and the job creation step in the CLI.

However, I suspect this 'additional step' will be less of an UX issue (perhaps even a net positive) for developers-as-broadcasters, because they would be more familiar with the mechanics that are leading to this choice, and they will probably appreciate the flexibility that is gained from this scheme.

"Time capped transcoding channel" is a good way to think of it. In fact, I've always thought of the job creation protocol as being a few small steps removed from state channels. I have a strong hunch we can take this concept even further, but that's for another proposal.

j0sh commented 6 years ago

One potential issue: a broadcaster could have several long-running, concurrent jobs on different transcoders. Since transcoders may take a while to submit claims, it's possible that a broadcaster's deposit could be overdrawn by a large margin, and the transcoder(s) wouldn't be able to receive the funds they are entitled to.

This is slightly less 'accidentally exploitable' now that we limit 1 job per node, but it's trivial to work around. There are various heuristics transcoders can apply in order to decide whether or not to transcode a segment (eg, maintaining a minimum average deposit per job, <user_deposit>/<outstanding_jobs_for_user> ), but these are far from foolproof. State channels begin to look attractive here.

j0sh commented 5 years ago

Fixed in https://github.com/livepeer/go-livepeer/pull/618