Move towards databricks-sdk

[DaanRademaker]

We should move towards databricks-sdk for all the api requests. This will make this project more maintainable in the future as we don't need to write the api calls to deploy databricks resources.

First let's merge this https://github.com/godatadriven/databricks-cdk/pull/1052 so we can create new branches for moving the different resources. I think best plan of attack is to create new pull requests for each resource, or atleast a seperate commit so we can track easily which have changed.

[DaanRademaker]

@dan1elt0m https://github.com/godatadriven/databricks-cdk/pull/1054

First pull request changing mlflow registry to new sdk

[DaanRademaker]

List of modules to check off before everything is moved to databricks-sdk. Some of these modules do not have unit tests. For each one that is changed unit tests should be created. Also prefer to have 1 pull request per change. We can tick off as we are getting these done.

account:

• [ ] credentials https://github.com/godatadriven/databricks-cdk/pull/1064
• [ ] networks
• [ ] storage_config
• [ ] workspace

cluster_policies:

• [ ] cluster_policy

clusters:

• [ ] cluster

dbfs:

• [ ] dbfs_file

groups:

• [ ] group

instance_pools:

• [ ] instance_pools

instance_profiles:

• [ ] instance_profile

jobs:

• [ ] job

mlflow:

• [x] experiment https://github.com/godatadriven/databricks-cdk/pull/1055
• [x] registered_model

permissions

• [ ] cluster_permissions
• [ ] cluster_policy_permissions
• [ ] experiment_permissions
• [ ] job_permissions
• [ ] models
• [ ] registered_model_permissions
• [ ] sql_warehouse_permission

scim:

• [ ] user

secrets:

• [ ] secret_scope
• [ ] secret

sql_warehouses:

• [ ] sql_warehouses

tokens:

• [ ] token

unity_catalog:

• [ ] catalogs
• [ ] external_storage
• [ ] metastore_assignment
• [ ] metastore
• [ ] permissions
• [ ] schemas
• [ ] storage_credentials
• [ ] volumes