It's already possible to enter a loop of repeated retries to connect to a Tenzir node. What's not yet working is that a lost connection gets transparently reestablished. We would like to add this capability transparently to the pipeline executor. The desired semantics are that multi-node pipelines can fail intermittently, without the user noticing it.
### Definition of Done
- [ ] Implement transparent reconnect in the pipeline operator with a configurable policy
- [ ] Policy A: simply drop newly arriving data
- [ ] Policy B: write incoming data into a fixed-size in-memory buffer
- [ ] Policy C: spill to disk (also fixed capacity)
It's already possible to enter a loop of repeated retries to connect to a Tenzir node. What's not yet working is that a lost connection gets transparently reestablished. We would like to add this capability transparently to the pipeline executor. The desired semantics are that multi-node pipelines can fail intermittently, without the user noticing it.