zenml-io / mlstacks

A series of Terraform based recipes to provision popular MLOps stacks on the cloud.
https://mlstacks.zenml.io/
Apache License 2.0
246 stars 32 forks source link

Error when deploying, and then importing, vertex-ai stack: `value is not a valid dict` #37

Closed tszumowski closed 1 year ago

tszumowski commented 1 year ago

Versions


gcloud --version     

Google Cloud SDK 419.0.0
alpha 2023.02.17
beta 2023.02.17
bq 2.0.85
core 2023.02.17
docker-credential-gcr 1.5.0
gcloud-crc32c 1.0.0
gke-gcloud-auth-plugin 0.4.0
gsutil 5.20
kubectl 1.24.10
zenml --version                             

zenml, version 0.34.0

Setup to Reproduce


I deployed the vertex-ai stack by following the readme.

zenml stack recipe pull vertex-ai
[edited locals.tf, gke.tf]
zenml stack recipe deploy vertex-ai --import

I edited locals.tf and gke.tf to allow us-east1 in the same manner the directions in the locals.tf specify for the europe region.

  # if you're using europe-west1, please make the following modification in
  # the gke.tf file:
  # For zones in module.gke, replace "${local.region}-a" to "${local.region}-d"
  # This is because "europe-west1-a" doesn't exist for some reason.

⬆️ that applies to us-east1 also.

Here are the two tf files I edited:

terraform_mods.zip

From there, I deployed it with:

zenml stack recipe deploy vertex-ai --import

Error


The stack deployment works fine. However when the import occurs, I get the following error:

ValidationError: 1 validation error for VertexOrchestratorConfig
labels
  value is not a valid dict (type=type_error.dict)

This is the full stack trace:

stack_trace.txt

Solution


This is how I solved it.

In the generated JSON file, there is:

    configuration: {"workload_service_account": "[SERVICE_ACCOUNT_EMAIL]", "project": "urbn-data-science", "location": "us-east1", "labels": "{}"}

I solved it by removing:

, "labels": "{}"
wjayesh commented 1 year ago

Hey Tom! Thanks for reporting the bug. The labels parameter was removed in recent versions of the Vertex orchestrator but this wasn't updated in the recipe. I'll push a PR to fix this :)