Introduce the new streamOutput field for pipeline stages.
Stream outputs will be required to be supplied as part of the pipeline ack payload, just as a stage output is required.
The payloads will be written to disk along side the pipeline stage's normal output, but will be nested under a separate prefix as {prefix}{offset}{stage_name}.
A notification will prompt the Stream controller that Pipeline Stream outputs need to be added to the Stream. It will transactionally copy the data to the Stream in proper order, preserving Stream controller encapsulation.
This will provide a more structured model for transactional processing overall.
Only a normal stage output can be used as input to another stage, not stream outputs.
Transactional Pipeline -> Stream outputs.
Introduce the new
streamOutputfield for pipeline stages.ackpayload, just as a stage output is required.{prefix}{offset}{stage_name}.