search

databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
https://dbx.readthedocs.io
Other
440 stars 120 forks source link

question #656

Open Arnold1 opened 1 year ago

Arnold1 commented 1 year ago

hi,

I have a python file of a tasks is called update.py - which is called by dbx. the python file calls the databricks api - which requires a DATABRICKS_TOKEN. how can I pass the DATABRICKS_TOKEN to the python file - which is part of a task of a workflow?

I already checked the docs - but was not able to find anything.

thanks

ptabandzelic commented 1 year ago

Well dbx is CLI tool, u export databricks token as environment variable export DATABRICKS_TOKEN= or configure it within .databrickscfg. https://docs.databricks.com/dev-tools/cli/index.html

Arnold1 commented 1 year ago

@ptabandzelic ok that sounds like a plan - at what stage to call the export DATABRICKS_TOKEN= ? before I call dbx deploy? what happens if the python file is called by the scheduler - is that env var still available? -> https://dbx.readthedocs.io/en/latest/reference/deployment/?h=or#scheduling-workflows

Arnold1 commented 1 year ago

I think another way would be that jinja support? https://dbx.readthedocs.io/en/latest/features/jinja_support/

ptaban commented 1 year ago

Bro i think u are confused.

renardeinside commented 1 year ago

hi @Arnold1 ,

Could you please explain a bit more in detail what you're doing?

Do I understand correctly that:

  1. You have a task in Databricks, the code of this task is in some_pkg/update.py
  2. Inside this task you would like to access the DATABRICKS_HOST and DATABRICKS_TOKEN variables to call specific Databricks APIs?
Arnold1 commented 1 year ago

hi @renardeinside

I want to create a one time run. I create a schedule for a specific time if the job runs I want to delete it.