Notebook task job spec - Githubissues

bweisberg commented 4 years ago

I need to write pipeline tests as Scala Notebooks. Will the databrickslabs_cicdtemplates Python module support more job specs?

for example:

  "notebook_task": {
    "notebook_path": "will get filled automatically"
  }

renardeinside commented 4 years ago

Hi @bweisberg!

Please take a look at the latest version, released today. The deployment configuration now also supports notebooks. Let me know if that is suitable for you.

chinwobble commented 3 years ago

@renardeinside I've looked through all the source code for dbx and I can't find any reference that supports executing or deploying notebook job tasks.

Could you elaborate on how to get this working?

renardeinside commented 3 years ago

Hi @chinwobble .

Notebook-based development is not a part of dbx (at least for now), so the execute functionality won't work. Also, we don't support (at least for now) the local-to-workspace notebook sync.

For the deployment part, specify the notebook spec in the deployment.json file. An example is here:

"jobs": [
 {
     "name": "my-notebook-job",

     // here goes your cluster spec etc

     "notebook_task": {
         "notebook_path": "/Users/my-username@example.com/workspace/path/to/notebook/notebook-name"
     }
 }
]

Here we use the REST API 2.0. Please find additional params for the notebook_task section here.

databrickslabs / cicd-templates

Notebook task job spec #18