Add Active Stack as part of configuration YAML

strickvl commented 7 months ago

Open Source Contributors Welcomed!

Please comment below if you would like to work on this issue!

Contact Details [Optional]

support@zenml.io

What happened?

After a discussion about the Run Config YAML and potential improvements, it became apparent that including the active stack name or ID in the configuration YAML would significantly enhance usability. This change would allow users to specify which stack the pipeline should run on directly within the config YAML.

Task Description

Update the ZenML configuration YAML structure to include an option for setting the active stack. Users should be able to specify the stack name or ID under a dedicated section in the YAML, which ZenML will then use to determine the active stack for pipeline execution.

Expected Outcome

The configuration YAML should have a clear and user-friendly section for specifying the active stack.
ZenML should correctly interpret and use the specified stack for running the pipeline.
This update will provide users with greater control and flexibility in their pipeline execution environment.

Steps to Implement

Design and implement a new section in the Run Config YAML format for specifying the active stack (name or ID).
Update the ZenML backend to read and apply this configuration when initializing pipeline runs.
Ensure backward compatibility and graceful handling when the stack section is not specified in existing YAMLs.
Test the implementation thoroughly to ensure the correct stack is used based on the YAML configuration. (Include tests in PR).
Update documentation to guide users on how to use this new feature in their YAML configurations.

Additional Context

This enhancement is part of ongoing efforts to make ZenML more flexible and user-centric, enabling users to tailor their MLOps workflows more precisely to their needs.

cc: @schustmi @znegrin

Code of Conduct

[ ] I agree to follow this project's Code of Conduct

kabinja commented 7 months ago

Hello, if that task is still available, I can start working on it :)

strickvl commented 7 months ago

@kabinja yes for sure! let us know if you have any questions, and please do follow the CONTRIBUTING document, esp w/r/t to branching and making your PR off the develop branch!

schustmi commented 7 months ago

@kabinja One little detail for you that will be quite helpful: This is the pydantic class that defines the structure of the yaml configuration file.

rahuja23 commented 1 month ago

Is this issue already resolved?

htahir1 commented 1 month ago

@rahuja23 no we're still waiting for some activity here.. I think @kabinja seems to have moved on from the task. If not, happy to pick it up from here / make another PR for it

htahir1 commented 1 month ago

There is a PR here that is stuck for the last 3 weeks https://github.com/zenml-io/zenml/pull/2370#issuecomment-2217234060

rahuja23 commented 1 month ago

So I can contribute on this issue then?

htahir1 commented 1 month ago

@rahuja23 yes i would say so, there is a PR already but its stale. You can start from there or from scratch

rahuja23 commented 1 month ago

@htahir1 Perfect thanks!

kabinja commented 1 month ago

Hello, sorry for not getting back earlier. I won't have much time this summer so please go ahead. If you have some questions about what I tried to do, please do not hesitate to reach out to me.

zenml-io / zenml