proposal: publish language-agnostic type definitions for stream profiles

[iameli]

Abstract

Let's publish a JSON-Schema for the body of the -authWebhookUrl repsonse.

Motivation

LivepeerJS has blocked release of new go-livepeer features because we have to update our API server every time to add new features — this happened with GOP length, H264 profiles, and fractional framerate support. We'd like to keep adding more video features and ship them accordingly. What if we published language agnostic type definitions that could be consumed by the API server and used to validate incoming API objects.

Another potential use case: a "rendition designer" front-end that would transform the schema into a visual interface (think Handbrake) that would allow you to pick from all the options supported by the schemas we'd publish.

Proposed Solution

I like JSON Schema, but I'm not married to it and would be open for any suitable intermediate format. I'd just like for it to be something that is widely adopted enough that we could use it to produce type definitions in Go, TypeScript, Java, maybe C++. From there, I see two ways we could go about it:

1. Write a JSON-schema (I already did but it would need improvement) and use that to generate the Go type definitions in the code, with a tool like atombender/go-json-schema.
2. Generate a JSON Schema from the existing go type definitions with a tool like alecthomas/jsonschema.

I like option one a bit more, as we could also use the schema to do run-time validation, which would allow for things like enum types to be enforced properly. But I'd be fine with doing either.

Implementation Tasks and Considerations

• Add logic to the Makefile that generates the build-time definitions
• Refactor existing code to use the new definitions when necessary
• In the CI process, publish the schema somewhere useful where it can be scraped by downstream projects.

Testing Tasks and Considerations

This would be a big DJ go-livepeer vs DJ LivepeerJS remix collabo project, so we'd probably want a pair of branches that could test against each other or something.

Known Unknowns

• Option 1 or option 2?

Alternatives

@adamsoffer do we have anything like this in place already on the crypto side of things? I know you generate a lot of type definitions from the protocol side... anything that would be useable for a project like this? I guess they could be published as Solidity types and then used on-chain for things?

Additional Context

[yondonfu]

Adding a few details that I think are useful to be aware of:

• The ffmpeg.VideoProfile Go type is defined in LPMS which is used by go-livepeer, but...
• The net.VideoProfile Go type is auto-generated by the protobuf compiler from a protobuf message defined in go-livepeer
• We use a helper function to convert between net.VideoProfile and ffmpeg.VideoProfile

So, an update to a JSON schema ala option 1 would need to:

• Trigger ffmpeg.VideoProfile type generation in LPMS
• Trigger a JSON -> protobuf message generation in go-livepeer to update the lp_rpc.proto file
• Trigger a protobuf message -> net.VideoProfile type generation in go-livepeer

And then after that a developer would still need to make sure functions that use the types in go-livepeer/LPMS are updated properly.

I wonder if it would make sense to just use the protobuf message in go-livepeer as the ultimate source of truth and generate language specific types from that since we already have it.

[iameli]

@yondonfu Unfortunately I don't think that would quite get the job done — the type needed by external webhook integrations is authWebhookResponse, which doesn't have precisely those same keys: https://github.com/livepeer/go-livepeer/blob/b0162eda798fe1c7b0eb62d5d2374bf3b25a7b5c/server/mediaserver.go#L97-L111

So that's got the string vs. integer GOP and whatnot. That said, publishing those protobuf specifications in the same place as the schema for that webhook could also be a good idea for potential future gRPC integrations for sure.

[j0sh]

Proxy mode makes API compatibility less urgent. Nodes can make use of new features immediately with the Livepeer ingest protocol, as long as production nodes are updated. That was one of the goals, after all - to decouple roll-out of node-first features from the API itself. Even if the schema definitions were shared, there'd still be coordination required around deployment of the node and the API, which may not always be easy to achieve.

With node-first, API compat would only be immediately blocking for a specific subset of users - those using the hosted RTMP service who need to configure new streams on the fly. Web UI users would also be blocked by updates to the UI itself.

The HTTP push protocol also technically has the same issue right now, but the whole scheme really should be deprecated by Livepeer ingest and a node-first approach to integration. HTTP push could also be modified to take stream configuration via header, but that's less desirable than transitioning to node-first.

With that in mind, my sense is that some duplication of the schema is fine. It's only a small amount of code and makes repo management a lot easier.