Closed robinholzi closed 3 months ago
GitHub could not determine anymore that merge conflicts were already resolved. The multi-stage merge conflict resolution and auto rebase likely bifurcated the git history. Getting rid of the commit history via a squash merge in #437
Motivation
The current state of
pipeline_executor.py
doesn't follow clear sequential or logical order. Also, the setup doesn't allow to automatically profile and log pipeline stage execution. Restructuring the orchestration of a pipeline in a PipelineExecutor here.Changes
PipelineExecutor
member functions and merge unused PipelineStages (likeRUN_TRAINING
andTRAINING_DONE
). This allows to have a 1:1 mapping between logical pipeline stages and member functions.@pipeline_stage(...)
decorator which wraps every pipeline stage function. When calling the decorated function we record meta information forlogging
(post-experiment analysis) andtracking
(for online feedback) purposes. This includesstart
&end
timestamps as well as the current "dataset/sample time". This is especially useful in experiments to be able to sort/plot different measurements by the time of the processed samples.grpc
) into a centralExecutionState
dataclass.pytest.mark.parameterize(...)
TODO
main