ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
If the following is nonsense, feel free to say so ;-)
Pipeline decorators for both the pipeline collocation and the pipeline components are classmethods. Effectively this saves the way the pipeline is built within a global singleton (here, the class). This couples the code tightly to a single, global object (the class). Not only is this a theoretical anti-pattern, but it practically hinders the possibility to have pipelines that call pipelines. I for one would like to build a hyperparameter optimization pipeline (which includes components like the actual HPO, a "built report" components, and managerial components) that calls the pipeline that does the actual training (which includes components like data preparation, training, and evaluation).
My sketch of a suggestion is to refactor this like so:
Turn all decorators into normal methods.
Create a class member P which instantiates the Pipeline class itself.
Create class members with the names of the previous classmethods and have them point at P's methods.
This way you could have your cake and eat it too, namely have normal methods and still have your current singleton behavior. I am not sure if my "sketch of a suggestion" is possible in Python, but maybe it is a starting point. In any case, I would like to suggest getting rid of the effective singleton.
If the following is nonsense, feel free to say so ;-)
Pipeline decorators for both the pipeline collocation and the pipeline components are classmethods. Effectively this saves the way the pipeline is built within a global singleton (here, the class). This couples the code tightly to a single, global object (the class). Not only is this a theoretical anti-pattern, but it practically hinders the possibility to have pipelines that call pipelines. I for one would like to build a hyperparameter optimization pipeline (which includes components like the actual HPO, a "built report" components, and managerial components) that calls the pipeline that does the actual training (which includes components like data preparation, training, and evaluation).
My sketch of a suggestion is to refactor this like so:
Pwhich instantiates thePipelineclass itself.P's methods.This way you could have your cake and eat it too, namely have normal methods and still have your current singleton behavior. I am not sure if my "sketch of a suggestion" is possible in Python, but maybe it is a starting point. In any case, I would like to suggest getting rid of the effective singleton.