Make node_type_id optional as it's otherwise impossible to create a cluster from resources in a pool

Description

Motivation and Context

This would help all teams that are using brickflow and want to run some of their jobs on clusters taking instance from cluster pools for performance, SLA or other reasons

How Has This Been Tested?

I succesfully deployed my workflow to Databricks where it made use of the instances provided by a pool. I ran 'make check' to align with the formatting standards. I ran 'make test' and everything was successfull

Screenshots (if appropriate):

When create cluster with instances from pool

Cluster(
   name="some_cluster_name,
   spark_version="any_spark_version",
   instance_pool_id="some_instance_pool_id",
  num_workers=1,
 )

and run brickflow deploy , on wheel containing my commits, it succeeds

[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {configure.py:callback:125} - Setting env var: BRICKFLOW_ENV to local...
[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__setattr__:159} - Configuring attr: BRICKFLOW_AUTO_ADD_LIBRARIES with value: True
[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {projects.py:use_project:296} - Changed to directory: /Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound
[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__setattr__:159} - Configuring attr: BRICKFLOW_PROJECT_RUNTIME_VERSION with value: 0.10.0
[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__setattr__:159} - Configuring attr: BRICKFLOW_ENABLE_PLUGINS with value: False
[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__setattr__:159} - Configuring attr: BRICKFLOW_PROJECT_NAME with value: trade-customs-emea-outbound
[2023-09-18 15:41:45,202] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__setattr__:159} - Configuring attr: BRICKFLOW_MONOREPO_PATH_TO_BUNDLE_ROOT with value: .
[2023-09-18 15:41:45,203] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {bundles.py:download_and_unzip_databricks_cli:157} - Databricks cli already exists. Skipping download.
[2023-09-18 15:41:45,203] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {commands.py:exec_command:25} - Executing command: .databricks/bin/cli/0.203.0/databricks --version
[2023-09-18 15:41:45,495] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {configure.py:log_important_versions:144} - Using bundle version: Databricks CLI v0.203.0
[2023-09-18 15:41:45,496] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {commands.py:exec_command:25} - Executing command: /Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/.venv/bin/python --version
[2023-09-18 15:41:45,506] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {configure.py:log_python_version:152} - Using python version: Python 3.9.5
[2023-09-18 15:41:45,506] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {bundles.py:bundle_synth:208} - Synthesizing bundle...
[2023-09-18 15:41:45,506] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {commands.py:exec_command:25} - Executing command: /Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/.venv/bin/python src/workflows/entrypoint.py
[2023-09-18 15:41:46,070] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__getattr__:147} - Getting attr: BRICKFLOW_PROJECT_NAME which has value: trade-customs-emea-outbound
[2023-09-18 15:41:46,070] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__getattr__:147} - Getting attr: BRICKFLOW_AUTO_ADD_LIBRARIES which has value: true
[2023-09-18 15:41:46,070] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {project.py:__post_init__:189} - Auto adding brickflow libraries...
[2023-09-18 15:41:46,070] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__getattr__:147} - Getting attr: BRICKFLOW_PROJECT_RUNTIME_VERSION which has value: 0.10.0
[2023-09-18 15:41:46,071] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {__init__.py:__getattr__:147} - Getting attr: BRICKFLOW_ENABLE_PLUGINS which has value: false
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/core/brickflow_utils.py:66: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  return ctx.dbutils_widget_get_or_else(
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/core/brickflow_utils.py:66: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  return ctx.dbutils_widget_get_or_else(
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/core/brickflow_utils.py:66: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  return ctx.dbutils_widget_get_or_else(
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/core/brickflow_utils.py:66: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  return ctx.dbutils_widget_get_or_else(
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/core/brickflow_utils.py:66: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  return ctx.dbutils_widget_get_or_else(
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/workflows/deploy_trade_customs_outbound_ddls_workflow.py:19: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  env = ctx.dbutils_widget_get_or_else("brickflow_env", BrickflowDefaultEnvs.TEST.value)
/Users/BBraec/Documents/GitHub/nike-glix/trade-customs-emea-outbound/src/core/brickflow_utils.py:66: DeprecationWarning: Call to deprecated function dbutils_widget_get_or_else.
  return ctx.dbutils_widget_get_or_else(
[2023-09-18 15:41:46,292] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {project.py:__exit__:289} - Deploying changes... to local
[2023-09-18 15:41:47,789] [INFO] [brickflow-framework-0.0.1rc23+fadf9bb] {commands.py:exec_command:25} - Executing command: .databricks/bin/cli/0.203.0/databricks bundle deploy -e trade-customs-emea-outbound-local
artifacts.whl.AutoDetect: Detecting Python wheel project...
artifacts.whl.AutoDetect: No Python wheel project found at bundle root folder
Starting upload of bundle files
Uploaded bundle files at /Users/Brend.Braeckmans@nike.com/.brickflow_bundles/trade-customs-emea-outbound/local/files!

Starting resource deployment
Resource deployment completed!

The same occurs when also specify driver_instance_pool_id

Cluster(
   name="some_cluster_name,
   spark_version="any_spark_version",
   instance_pool_id="worker_instance_pool_id",
   driver_instance_pool_id="driver_instance_pool_id", 
  num_workers=1,
 )

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Nike-Inc / brickflow