Resolve Vertex Scheduler Character Limit Issue for Long Pipeline Names

strickvl commented 7 months ago

Open Source Contributors Welcomed!

Please comment below if you would like to work on this issue!

Contact Details [Optional]

support@zenml.io

What happened?

A user encountered an issue where the Vertex scheduler in ZenML does not accept pipeline names longer than 64 characters. This limitation requires users to shorten their pipeline names to successfully schedule their pipelines on Vertex AI, which can be restrictive and inconvenient.

Task Description

Investigate and address the issue with the Vertex scheduler's character limit for pipeline names in ZenML. The goal is to either increase the character limit or implement a mechanism that automatically handles longer names without user intervention.

Expected Outcome

Users should be able to schedule pipelines on Vertex AI without the constraint of a 64-character limit on pipeline names. ZenML should provide a seamless experience, either by accommodating longer names or by intelligently managing them within the existing Vertex scheduler limitations. This fix will enhance user convenience and flexibility in naming pipelines.

Steps to Implement

Reproduce the issue to understand the exact limitations and behavior of the Vertex scheduler with respect to pipeline names.
Explore solutions such as increasing the character limit or automatically abbreviating/transforming long pipeline names while preserving their uniqueness and recognizability.
Implement the chosen solution in a way that is transparent and intuitive for the user.
Test the implementation with various pipeline name lengths to ensure compatibility and functionality.
Update documentation to inform users about the handling of pipeline names in Vertex scheduler and any best practices or limitations.

Additional Context

This issue is crucial for users who require descriptive and potentially lengthy names for their pipelines, especially in complex or large-scale MLOps environments.

Code of Conduct

[ ] I agree to follow this project's Code of Conduct

strickvl commented 7 months ago

@christianversloot if I'm not completely mistaken, we had a conversation about this in Slack? Unfortunately the slack history hasn't been retained.

christianversloot commented 7 months ago

Correct, but that was for SageMaker orchestrator. There, I believe we limit name length to circumvent this error. It could perhaps be inspiration for a fix here :)

strickvl commented 7 months ago

@christianversloot ah yes! https://github.com/zenml-io/zenml/pull/1505 was the PR.

ashutosh887 commented 7 months ago

@strickvl @christianversloot I would like to work on this... Please assign!

Merthoshan commented 7 months ago

has the bug been solved? if not i would like to solve it

strickvl commented 7 months ago

The bug has not been solved yet @Merthoshan. @ashutosh887 are you working on it, if not, can let @Merthoshan take a shot.

ashutosh887 commented 7 months ago

Where can we discuss if we're facing some issues understanding the bug? @strickvl I've worked it

strickvl commented 7 months ago

Here is the place.

zenml-io / zenml