search

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
202 stars 100 forks source link

Problem running dbx execute with example #28

Closed dexianta closed 3 years ago

dexianta commented 3 years ago

I'm not able to run dbx execute --cluster-name my-cluster --job job-sample. As far as I understand, in dbx execute, wheel was built, copied to dbfs, and jobs/sample/entrypoint.py was serialized to directly run on the interactive cluster, without providing --conf-file. However the Job class defined in project_name/common.py requires --conf-file, thus exception were thrown image

======================================== So far, it seems to me that:

renardeinside commented 3 years ago

dbx execute is suitable to launch script without external config files, since it's using 1.2 API command execute.

Yes, you're correct in this understanding.

if I want to run the local code on interactive cluster (for fast dev iteration), I'll need a separate job json with "exisiting_cluster_id". then run launch (which is /job/run-now underneath)

I'll provide the functionality to add a default_conf method, which allows you to provide a default configuration inside code for dbx execute-based dev.

renardeinside commented 3 years ago

@dexianta please check the latest version and especially the init_adapter method.

dexianta commented 3 years ago

@dexianta please check the latest version and especially the init_adapter method.

@renardeinside thank you very much