[FEATURE] Enable fleet instances for `node_type_id` when cluster creation

Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow

https://engineering.nike.com/brickflow/

Apache License 2.0

183 stars 36 forks source link

[FEATURE] Enable fleet instances for `node_type_id` when cluster creation #100

Closed asingamaneni closed 4 months ago

asingamaneni commented 5 months ago

Is your feature request related to a problem? Please describe. Right now node_type_id is not supporting fleet instances. May be we need to update the cli and see if DAB's support it.

Cloud Information

[x] AWS
[ ] Azure
[ ] GCP
[ ] Other

Describe the solution you'd like Update the Databricks cli

Describe alternatives you've considered NA

Additional context Add any other context or screenshots about the feature request here.

asingamaneni commented 4 months ago

It already is working good and able to deploy

Tried with below config