RFC: Picking future technologies

nytamin commented 2 months ago

About me

This RFC in on the behalf of NRK

Use case

For quite some time now, there have been discussions in the NRK Sofie team as well as with other contributors to Sofie on whether we should move away from (parts of) the current technology stack and libraries. We think that now is a good time to initiate those discussions.

We're not really looking at replacing the whole of Sofie with something completely different, but rather look at the individual components that Sofie consists of and evaluate whether they should be replaced with something else.

It is of our opinion that we should move away from the Meteor framework, due to it’s stalling development and it no longer being a good fit for us (tooling is scarce, attaching a debugger works different to regular Node, bundling is slow and Hot Module Reloading support is poor, it makes too many decisions for us and picks libraries that we don’t like).

Proposal

We invite all Sofie contributors to a series of workshops where we share our thoughts and experiences with various technologies/libraries.

When discussing technologies, we think that it'll be important to consider

Developer experience (do we like it?)
Future maintenance and reliability (is the tech mature, maintained, commonly used?)
Transition period (does replacing the component affect other parts of Sofie ecosystem? Can it be implemented and used in parallel with existing tech, to not delay release cycles?)
Implementation cost (ie time needed to replace the component)

Technology areas

I've listed various technology areas/components that we might want to discuss below, feel free to suggest other areas as well and I’ll add them to the list.

Frontend/GUI server

Currently, we're using Meteor's built in dev server. We'd like something that offers a better developer experience, most notably hot module replacement.

During the transition, we could eject the GUI from Meteor into another mono-repo package and serve it from there during dev. Then, after bundling for production, let Meteor serve in prod.

Possible options are Vite (seems to be the go-to solution for new projects in 2024), Webpack (established option, lot’s of documentation and tooling), others...?

GUI state management

Currently, we’re using Meteor’s minimongo pseudo-mongodb.

During the transition, we could implement the state management and let minimongo feed it with input data and rewrite the React components independently.

Examples of libraries: mobx, redux, xstate, zustand, others…?

GUI render framework

Currently, we’re using React. We’re happy with it.

Examples of libraries: Vue, Angular, preact…

Data/API pipe

Currently, we’re mainly using Meteor’s DDP-protocol for pub/sub, for our GUI and Gateways. Then there’s also websocket with asyncapi in the Live Status Gateway and REST with openapi for the “stable API”.

Examples of libraries: GraphQL, Feather.js, >>Koa/feathers/raw implementation<<

Backend apps / messaging service

We don’t have a messaging service at all the moment, our job-workers talk directly with the main process.

Perhaps we should consider moving the job-workers to separate containers and use a backend messaging service?

Examples of technologies: RabbitMQ, Kafka, NATS..

Backend processes / workers

Currently, the backend consists of 4 processes:

The main process (runs Meteor, serves the GUI, handles pub/sub to GUI and gateways, interfaces with the workers)
Parent (handles IPC and spins up the other workers)
Ingest (handles data from the NRCS, runs blueprints to generate the Rundown)
Studio (handles playout, generates the timeline)
Events (handles events, like sending asRun messages)

Should we change this setup, the number of processes/apps and/or their responsibilities?

Multithreading in TSR/workers

Currently we’re relying on threadedClass library, which is sometimes fiddly to use.

Alternatives to using threadedClass?

Persistant store / Database

We’re currently using MongoDB as backend, we’re pretty happy with that.

If you’re interested in joining the discussion, please add a reply to this thread or send me a message at johan.nyman@nrk.no . Let me know which areas you are particularly interested in discussing.

Process

The Sofie Team will evaluate this RFC and open up a discussion about it, usually within a week.

[x] RFC created
[ ] A workshop has been planned
[ ] RFC has been discussed in a workshop
[ ] A conclusion has been reached, see comments in thread

PeterC89 commented 2 months ago

Definitely interested in joining discussion 🙂

I don't have strong opinions about which specific technology choices to use but I would like to strongly argue in favour of technology choices that make it easy to manage in a distributed (Kubernetes like) environment where you can have multiple instances of Sofie running for redundancy as opposed to the one of everything approach that the current arrangement restricts you to. As well as moving to a Data/API Pipe that works well across multiple languages (I.E. I can write a gateway in .net and not have to re-invent the DDP library in that language).

How very exciting!

ianshade commented 2 months ago

I'll gladly participate on behalf of TV 2 Norge. As LSG and Stable API users we're mostly interested in the future of how Sofie can be interacted with from the outside, and how its data can be obtained in real-time (the Data/API pipe). GUI state management is also a topic where we could exchange insights.

nytamin commented 2 months ago

I've planned a workshop on Tuesday, May 7th at 15:00 CEST (that's 14:00 BST, 9am EST) and sent out invites to the ones who have shown interest. Notes from the workshop will be posted in this thread.

If anyone else wishes to join the workshop, please email me at johan.nyman@nrk.no

nytamin commented 1 month ago

Notes from workshop 2024-05-07

Frontend/GUI server/bundlers

Webpack is a proven work horse, but seems to be quite heavy and slow during dev.
Vite has been a good experience in 2 recent NRK projects, it offers a fast development experience. Possible issue; usage of local commonjs modules in workspaces (need to be investigated).
Rollup, not much experience
Parcel, good for small projects, but is it expandable enough?

Our recommendation would be to go for Vite, or possibly Webpack.

GUI state management

mobx: Is generally liked by us. It is less opinionated than redux, which could be both good or bad.
redux: Is a more strict approach to state management than mobx. With tooling, like Redux toolkit, development is easy. With normalized data, performance can be very good.

No one had any experience with other alternatives.

GUI render framework

Recommendation: Continue with React (no one had any objections).

Data/API pipe

GraphQL: Isn’t primarly a realtime api, its competing with REST. Probably not that suitable for us.
Feather.js: Is liked by the team. One drawback is that it is not that well used, should we go for something more popular?
Socket.io: Very popular, but doesnt provide much high-level functionality.
asyncapi: Lacking in good generators (for nodejs at least), but is that a dealbreaker?

Open question: Is there any library that handles data-syncing across clusters or in cases of reconnections?

We ran out of time at this point.

Next workshop will be Tuesday, May 14th at 15:00 CEST (that's 14:00 BST, 9am EST). If you want to join the discussion, shoot me an email at johan.nyman@nrk.no to get an invite.

nrkno / sofie-core