Forked Pipelines

[dominiklohmann]

Pipelines currently always run in the same process as the node when run through the API. This is a reliability problem, because a pipeline running out of memory also causes the node to go down, and with it all other pipelines.

This process is about decoupling the risk of running pipelines by running them in a child process instead.

### Stories
- [ ] https://github.com/tenzir/issues/issues/1547
- [ ] https://github.com/tenzir/issues/issues/745

[lava]

I think one big thing to keep in mind here is that this turns the tenzir process essentially into a process manager. So we have to be careful about keeping track of all these forked processes we spawn, and how to clean them up again on shutdown.

[mavam]

Our friends at Zeek put a lot of energy into getting process supervision right. It may make sense to study the supervisor framework at a very high level to avoid re-experiencing all the weird POSIX gotchas.

[lava]

We should also take a close look at a deeper cgroup integration of the node, after all this kind of scenario (ie. bounding memory/cpu usage of a group of processes and not losing track of them) is what they were invented for.

[dominiklohmann]

This came up again today when preparing for a demo. It's a real bummer to have a node crash because of a bug in a third-party library used in a connector. That sometimes is just out of our control.

[lava]

On Wednesday we had a discussion round together with @dominiklohmann and @jachris .

We agreed on the high-level outlines of this feature:

• We want to implement new logic for pipelines spawned via REST endpoint, so that the local for these pipelines refers to a newly spawned process
• The executor for these forked pipelines will still be running inside the node by default
• If easily possible we want to use cgroups to track the spawned processes, but if the overhead is high we will prioritize development speed and a mostly-working solution over robustness
• This new forked mode will be enabled by default (maybe controlled by the node config). We want to provide an opt-out for high-performance pipelines (ie. pcap direct capture) where ipc overhead is noticeable, together with a frontend way to control this, but it's out of scope for this roadmap item
• Persistent pipelines (ie. surviving node crashes), automatic reconnects, kill policies, resource limitations, etc. are out of scope for this roadmap item
• Authenticating/securing the connection between node and pipeline is out of scope for this roadmap item (we may be able to make use of the upcoming caf domain sockets support for this)

Related documents: https://docs.google.com/document/d/1b-zpDp796fRr1FPpObCkia2Dyuh8IEF-XaBmv5lvszs/edit#heading=h.um2utrvlnup8 https://app.excalidraw.com/s/6dBWEFf9h1l/8J1RozwXFXV

[tobim]

I attempted to write a proof of concept for this last night / this morning and got to a point where I can run a pipeline in a forked tenzir-node process as a whole. The core behavior change of reducing the blast radius of a crashing pipeline would be fulfilled by that. But the majority of the work is yet to be done. Notably:

• Properly integrating with the operator location semantics.
• Either reconnecting to the parent node in case it restarted or going down with it (both options are tricky to implement).
• Maybe doing this in the pipeline executor directly to avoid unnecessary ipc of the metrics of remote operators.
• Testing and proper error handling.