Open hmh opened 5 years ago
Implementation ready, it is undergoing testing and we will submit it as a PR soon (it doesn't depend on any pending PRs).
One border condition: if a schedule has no actions, it currently will accumulate input in its storage (because it doesn't actually "run"). We have not changed that behavior in our soon-to-be-submited PR, we basically considered it undefined for now.
On sequential schedules, it may be useful to allow one action to act on the output of a previous action (sort of like the still not-implemented pipelined mode, but not really).
We have special cased an action that has its own schedule listed as its output destination, so that the output is placed directly into that schedule's processing queue and thus it is immediately available for the next action.
Without the special casing, the output would be directed to the incoming queue of the schedule, and would be available only on the next execution of the whole schedule.
The current data flow from an action (in schedule A) to destination(s) in schedule B... is:
This really gets in the way of implementing resilient reporting of results to a collector. The desired report flow is: render the report, optionally compress it, and if this fails (e.g. storage full), don't remove the source data. After the report is rendered successfully, remove only the source data that is present in that report, and queue the report for transmission. The report is only removed when the transmission succeeds. The render+transmit steps may be implemented atomically, and then it becomes "the source data present in the report must only be removed when the transmission succeeds".
Proposed action -> schedule data-flow:
The SIMET team will implement the proposed action above, and it will (as usual) be made available for merge upstream in our fork. If we find any problems with the proposed solution, we will edit this issue report accordingly.