databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.
https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html
Apache License 2.0
416 stars 141 forks source link

Error at deploying new model #143

Closed Eiley2 closed 6 months ago

Eiley2 commented 7 months ago

Hi, it's me again!

I'm currently facing an issue where whenever I change the model name, it gives me this error:

What I'm changing:

bundle:
  name: age

variables:
  experiment_name:
    description: Experiment name for the model training.
    default: /Users/${workspace.current_user.userName}/${bundle.target}-${var.model_name}-experiment
  model_name:
    description: Model name for the model training.
    default: age-model

The error

Deploying resources...
Updating deployment state...
Error: terraform apply: exit status 1

Error: unknown is not fully supported yet

  with databricks_grants.registered_model_model,
  on bundle.tf.json line 25, in resource.databricks_grants.registered_model_model:
  25:       }

If I use my old model who is already deploy with the bundle, it works perfectly. I'm using the other repo as a base, theorically it should work the same but with another name am I right?

arpitjasa-db commented 7 months ago

Hi @Eiley2, yup the other name shouldn't be a problem. Usually we see this error when there's a naming conflict. Can you confirm the name isn't overlapping with the other targets/catalogs and it doesn't already exist?

If there's no overlap, can you do a quick global search on the directory for instances of the old name and see if there's anywhere that needs to be changed still?

Eiley2 commented 7 months ago

I just did that, and there's no references to the old model in the assets files or databricks.yml file.

This how I'm named the models, should be it?

image

Also this is the databricks.yml

image

arpitjasa-db commented 7 months ago

Hmm I don't see anything wrong with the screenshots you've provided, but is there a reason why you don't update the model_name in databricks.yml to ${bundle.target}-age-model there directly? Mainly, there are other workflows that will directly use ${var.model_name} that won't need to be updated if you do it this way.

But in terms of the error that you're seeing, are you deploying this from the CI/CD or locally via CLI?

Eiley2 commented 7 months ago

Nice suggestion! I will implement it right away!

Both, for testing purposes I'm currently using my CLI, but I use the same token api for both just in case something works different with my admin user.

arpitjasa-db commented 7 months ago

Hmm sometimes the local state (.databricks) folder gets into an inconsistent state, so I was going to suggest moving that or deleting it. But if it's also happening with CI/CD, that may not be the case. Still may be worth a shot, otherwise I think there is something funky going on with the catalog/schema in the Databricks workspace. One more idea could be to simply remove the grants field temporarily and see if the error changes or goes away

Eiley2 commented 6 months ago

I guess something weird was happening with the CLI because I created a brand new project and implemented the stuff that I need and it worked, idk why. Thanks for the help as always!