RFC: Ingest API over HTTP

[sbaudlr]

This issue has been opened by SuperFly.tv on behalf of EVS Broadcast Equipment (henceforth EVS).

We propose to extend the Sofie API to present an ingest API to allow ingest data to be sent to Sofie from external sources via HTTP rather than DDP.

Motivation

There is a desire for an external system to be able to "push" data to Sofie via an API. As the HTTP API has been used in other areas of EVS work, it is desirable to also have this external system use the HTTP API.

Proposal

A draft of the shape of the API has been published and a PR has been opened.

There are a couple of patterns to point out in this draft. The first is the use of ETags and If-None-Match, the second is that the proposed API allows for external Ids to be used interchangeably with Sofie-internal Ids. Both patterns are explored below.

ETags and If-None-Match

The ETag and If-None-Match HTTP headers have been included as part of this proposal as a way of identifying the version of each ingest item (e.g. Rundown, Segment) when it is pushed to Sofie. Using these headers means that an external system can provide us a version string for some ingest data via the ETag and tell us to only update an existing item if it has a different ETag to the one provided. This means that we have a chance of defending against too many updates being sent by an external system, so long as the external system provides us with a sensible ETag to use as the version of the data (e.g. timestamp that the data was last modified).

Using this method also allows the entire API to be expressed as PUTs while still allowing for "forced" updates - e.g. in the case of re-syncing data. We achieve this by specifying that any update that is not accompanied by an If-None-Match header is always treated as a "create" event rather than an update, meaning that we will overwrite any existing data in this case.

This approach also allows for some performance improvements to be made in some cases, as we will be able to discard an update without performing any costly work within blueprints if we have received data that has the same version string as the object already in the database.

However, to use this version information we will need to store it somewhere which means that this proposed change does mean a change to the data present in the ingest data collections in the form of an optional version: string property. Also of note is the fact that bulk updates (e.g. a PUT for a list of Rundowns) will use If-None-Match to optimize bulk updates, but will not accept an ETag for these updates, so the next update to each individual element within a collection will be treated as a create. This is in-line with the recommended usage of ETag. The effect of this is likely to be that we will be able to optimize the initial ingest of data at the expense of unnecessary updates in the future when script edits are carried out on individual elements of the show.

Interchangeable Ids

In the current proposal, external systems can make calls to the Ingest API using either the Id internal to Sofie's database, or using an arbitrary external Id (so long as this Id is sufficiently unique). This allows the external system to interact with Sofie purely through its own Ids, without having to perform lookups to Sofie's API to maintain its own map between Ids. This also means that we can express the entire API as a series of PUTs rather than needing data to be created via a POST initially, which reduces the need for the external system to have an awareness of the data that is currently held within Sofie and by extension reduces the amount of knowledge the external system needs to have about the lifecycle of Sofie.

The cost of this approach should be minimal as it can be implemented using the $or operator in Mongo queries. Also, Ingest operations are generally not so time-sensitive that this lookup will cause excessive delay.

It may be useful to extend this approach to other areas of the API but that is outside the scope of this RFC.

Other things of note

The design of this API is such that the ingest objects contain a generic payload object, the exact shape of which is left to be agreed between the blueprints authors and the authors of the system that is pushing data to Sofie - in much the same way that the existing ingest gateways work. This also follows the pattern established by the Studio and ShowStyle config APIs, so anything that is developed in terms of validation and schema discovery for those methods can be shared with this API.

As part of the Playlist ingest objects, a resyncURL can optionally be provided that will be used by the "Reload data" action within the Rundown UI. This URL will receive a POST with an empty body to prompt an external system to resync data with Sofie, which is described as a request for all data for a Playlist to be re-sent to Sofie without the If-None-Match header present.

Request For Comment

• Does this make sense to expose over the HTTP API?
• Does the proposed shape make sense?
• Are we happy to employ this pattern of mixing external Ids with Sofie-internal Ids?

[jstarpl]

So here are my very hot takes:

• I think there is a general sentiment that we need to orient ourselves towards removing DDP as an integration protocol (at the very least), so I think having an HTTP ingest protocol is a good idea
• That being said, I think having an way of doing ingest via HTTP requires us to map the PeripheralDevices API onto HTTP somehow. There's a couple of things to think about there:
  • Authentication
  • Configuration
  • Status
  • Callbacks (such as "Reload from NRCS" action in case of Ingest devices)
• I don't think we should create an API only to then rip it out once we settle on - what we already know - is needed, so I personally would prefer that going this route would include settling on an API for HTTP-based PeripheralDevices, which I think is a worthy goal.

Now, for authentication, I can easily see Sofie going in the way of OAuth2 client credential workflow for PeripheralDevices, or going with a Authorization Code Flow for situations in which the Peripheral Device can act on behalf of a user. For callbacks, I think we could investigate something like GitHub's webhook system, which GitHub apps can automatically configure: decide what events they want to subscribe to, expose what callbacks they support, etc. However, what I can't immediately picture is what the Status reporting and centralized Configuration would look like in this API.

NRK's Sofie team has a "Developer's meeting" every Friday, I'll bring this RFC up, so that we can produce a fully-baked position.

[Julusian]

For those who aren't familiar with the current ddp api that this will be an alternative to: https://github.com/nrkno/sofie-core/blob/a72d868da712b32b3c0c81fcf66c76cd02274ece/packages/shared-lib/src/peripheralDevice/methodsAPI.ts#L79-L144 With the Ingest* types defined as: https://github.com/nrkno/sofie-core/blob/master/packages/blueprints-integration/src/ingest.ts These Ingest* types are also the ones that we provide the blueprints when performing ingest operations.

---

Interchangeable Ids

Do we need to do it like this? As far as I'm aware, the current generic ingest api over ddp solely uses the externalId types and we don't even provide back the actual document ids. We did this for precisely the same reason, so that the gateway can use the ids as defined in the system it gets the rundowns from.

So my question here is does the external system need to know the sofie ids? Is it wanting to use these to match up the rundowns/playlists with data retrieved from the other apis or live-status-gateway?

---

I haven't given the yaml a proper read yet.
One major difference I have noticed is that you are wanting to be able to define the playlists over this api. I am not sure if this will work, as this goes against the current model of how playlists are managed in sofie. We currently expect it to be the blueprints who define the playlistExternalId as part of ingesting a rundown. This is done because MOS and our other ingest data sources don't have a concept of playlists, this was something we devised to be able to stitch together multiple rundowns. It could be that what are doing with NRK should be more built into sofie, if it would be beneficial I could work out what exactly we need and we can look at whether it would fit elsewhere for this.

Have you thought about whether it will be possible to have these two systems in tandem?

Do you need to have playlists in the api, or was this added for completeness?

[jstarpl]

So, here are our notes after the developers meeting, on the general sentiment of the NRK team:

• We would like to have a roadmap for this feature, before we bring it into the fold. The REST API so far has been "remote buttons" for the system, executing actions that were otherwise available in the GUI anyway. This change would expose what used to be tied into the system through the server-integration package and make a "public" Ingest API. As such, since we've made a pinky-promise to keep the Stable API "Stable", unless otherwise neccessary, we would like to make sure that this API is reasonably complete and will be in fact "Stable" to the point where non-Sofie related systems can integrate it into their code.
• Having a HTTP-only protocol for Peripheral Devices would be nice, and we've brainstormed a couple of solutions that could potentially enable feature-parity with DDP but still offer a graceful degradation to a "simple curl client". This would probably include some sort of a token for authentication and maybe an "upgrade" of a WebSocket connection for status reporting, pings and interactive configuration, allowing us to mirror DDP-based PeripheralDevices in functionality.
• Ideally, the new HTTP ingest protocol should enable one to create various ingest gateways and combine the "generic" input gateway methods and the MOS-specific input methods into a single, flexible system
• We think that this API, considering that it's supposed to be the "ingest" interface, should still maintain little knowledge about Sofie's internals. As such, we're unsure about including the knowledge of "Studios" and "Playlists", or Sofie internal IDs, since these are Sofie internal inventions.