Resolve Vertex Scheduler Character Limit Issue for Long Pipeline Names

strickvl commented 10 months ago

Open Source Contributors Welcomed!

Please comment below if you would like to work on this issue!

Contact Details [Optional]

support@zenml.io

What happened?

A user encountered an issue where the Vertex scheduler in ZenML does not accept pipeline names longer than 64 characters. This limitation requires users to shorten their pipeline names to successfully schedule their pipelines on Vertex AI, which can be restrictive and inconvenient.

Task Description

Investigate and address the issue with the Vertex scheduler's character limit for pipeline names in ZenML. The goal is to either increase the character limit or implement a mechanism that automatically handles longer names without user intervention.

Expected Outcome

Users should be able to schedule pipelines on Vertex AI without the constraint of a 64-character limit on pipeline names. ZenML should provide a seamless experience, either by accommodating longer names or by intelligently managing them within the existing Vertex scheduler limitations. This fix will enhance user convenience and flexibility in naming pipelines.

Steps to Implement

Reproduce the issue to understand the exact limitations and behavior of the Vertex scheduler with respect to pipeline names.
Explore solutions such as increasing the character limit or automatically abbreviating/transforming long pipeline names while preserving their uniqueness and recognizability.
Implement the chosen solution in a way that is transparent and intuitive for the user.
Test the implementation with various pipeline name lengths to ensure compatibility and functionality.
Update documentation to inform users about the handling of pipeline names in Vertex scheduler and any best practices or limitations.

Additional Context

This issue is crucial for users who require descriptive and potentially lengthy names for their pipelines, especially in complex or large-scale MLOps environments.

Code of Conduct

[ ] I agree to follow this project's Code of Conduct

strickvl commented 10 months ago

@christianversloot if I'm not completely mistaken, we had a conversation about this in Slack? Unfortunately the slack history hasn't been retained.

christianversloot commented 10 months ago

Correct, but that was for SageMaker orchestrator. There, I believe we limit name length to circumvent this error. It could perhaps be inspiration for a fix here :)

strickvl commented 10 months ago

@christianversloot ah yes! https://github.com/zenml-io/zenml/pull/1505 was the PR.

ashutosh887 commented 10 months ago

@strickvl @christianversloot I would like to work on this... Please assign!

Merthoshan commented 10 months ago

has the bug been solved? if not i would like to solve it

strickvl commented 10 months ago

The bug has not been solved yet @Merthoshan. @ashutosh887 are you working on it, if not, can let @Merthoshan take a shot.

ashutosh887 commented 10 months ago

Where can we discuss if we're facing some issues understanding the bug? @strickvl I've worked it

strickvl commented 10 months ago

Here is the place.

Prabhat-Thapa45 commented 2 months ago

is there a simpler way to replicate this I could take this but It's going to be difficult for me to replicate this if this needs me to setup GCP which requires some cost. @strickvl

zenml-io / zenml