Closed di-lin-mckinsey closed 2 years ago
Hi @di-lin-mckinsey ,
Note this line in the setup:
Configure Databricks CLI connection profile
The project is designed to use 3 different Databricks CLI connection profiles: dev, staging and prod. These profiles are set in e2e-mlops/.dbx/project.json. Note that for demo purposes we use the same connection profile for each of the 3 environments. In practice each profile would correspond to separate dev, staging and prod Databricks workspaces. This project.json file will have to be adjusted accordingly to the connection profiles a user has configured on their local machine.
Given that running dbx deploy
from your local machine will use the local dbx
profile you will need to have this configured correctly. See [Databricks CLI connection profile](https://docs.databricks.com/dev-tools/cli/index.html#connection-profiles on how to do this. If your connection profile is called my_profile
, then you will have to replace e2-demo-west
with my_profile
in e2e-mlops/.dbx/project.json.
Let me know if you are still running into issues after doing this
Hi @niall-turbitt , thanks for your comment. I did follow the instruction to set up my connection profile. And I set my profile name as e2-demo-west
to advoid changing in the project.json
file. Here is the testing of my profile connection.
Sine the error mentioned "Job telco-churn-initial-model-train-register was requested, but not provided in deployment file", I wonder where the deployment file is and if the job telco-churn-initial-model-train-register
is provided there.
I see. In the deployment.yml
file, the job name is PROD-telco-churn-initial-model-train-register
. So should the right command be dbx deploy --jobs=PROD-telco-churn-initial-model-train-register --environment=prod --files-only
?
ah good catch, that is indeed a typo in the README.md. Did the following then work for you?
dbx deploy --jobs=PROD-telco-churn-initial-model-train-register --environment=prod --files-only
I've just updated the README to reflect the changes in the names of the jobs being deployed. I had updated the job names to reflect the environment the job was deployed to. This was to make the job names more distinct from one another if operating in a scenario where you had a single workspace with dev/stage/prod partitioned by ACLs. Please let me know if you run into anything else!
Hi @niall-turbitt , yes, dbx deploy --jobs=PROD-telco-churn-initial-model-train-register --environment=prod --files-only
works fine for me. I have also set up 3 different workspaces for dev, staging and prod with separate profiles for the the deployment to work in an actual setting. However, if you read the other issue I opened, I passed the deploy stage but failed again at the launch stage. Would you offer any insights there? Thanks.
Good to hear. Will close this issue. I've just replied to the other issue as well
Hi,
I'm following the tutorial to run an e2e mlops pipeline. After setting up my databricks CLI and github secrets, I ran the first workflow step:
dbx deploy --jobs=telco-churn-initial-model-train-register --environment=prod --files-only dbx launch --job=telco-churn-initial-model-train-register --environment=prod --as-run-submit --trace
.However, the deployment failed with following error.
May I know if I have missed any step in between? Thank you.