search

databricks / cli

Databricks CLI
Other
147 stars 55 forks source link

Failed to marshal state to json: unsupported attribute "ebs_volume_iops" #1447

Closed Ribeiro-Breno closed 3 months ago

Ribeiro-Breno commented 5 months ago

Describe the issue

With the release of databricks-cli 0.219.0 we started to face an issue with databricks bundle run ("panic error") and to avoid it we were using the previous version, 0.218.1. Now the 0.220.0 is released, the "panic error" remains and we have another error with version 0.218.1: "Failed to marshal state to json: unsupported attribute "ebs_volume_iops""

Configuration

All bundles using databricks-cli v 0.218.1 (the repos that don't have the "panic error" work with 0.220.0).

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

  1. Run databricks bundle run using databricks-cli 0.218.1

Expected Behavior

The job should run

Actual Behavior

The job is not started and the error is raised

OS and CLI version

CLI v 0.218.1

Is this a regression?

No

Debug Logs

{
  "variables": {
    "index_resolution": {
      "default": "10",
      "value": "10"
    },
    "na_job_cluster_policy_id": {
      "value": "0004D5D665AAB8E2",
      "lookup": {
        "cluster_policy": "NA-GADP-BPI_DEV_job"
      }
    },
    "output_catalog": {
      "default": "gadp_scratch",
      "value": "gadp_scratch"
    },
    "output_schema": {
      "default": "cdm_mapping",
      "value": "cdm_mapping"
    },
    "shapefile_url": {
      "default": "test",
      "value": "test"
    },
    "zip_url": {
      "default": "test",
      "value": "test"
    }
  },
  "bundle": {
    "name": "cdm_dsci_adas_geospatial",
    "target": "staging_na",
    "environment": "staging_na",
    "terraform": {
      "exec_path": "/home/runner/work/cdm_dsci_adas_geospatial/cdm_dsci_adas_geospatial/.databricks/bundle/staging_na/bin/terraform"
    },
    "git": {
      "branch": "dev/bundle_validate",
      "origin_url": "https://github.com/StellantisGADP/cdm_dsci_adas_geospatial",
      "commit": "47b9b999c45bca6cdad3581a0dc5e791f7a93cab",
      "bundle_root_path": ".",
      "inferred": true,
      "actual_branch": "dev/bundle_validate"
    },
    "mode": "development",
    "deployment": {
      "lock": {
        "enabled": false
      }
    }
  },
  "include": [
    "jobs/job.yml"
  ],
  "workspace": {
    "host": "https://dbc-deebf286-07d0.cloud.databricks.com",
    "current_user": {
      "active": true,
      "displayName": "gadp-bpi-cdm-dbx-sp-na",
      "emails": [
        {
          "primary": true,
          "type": "work",
          "value": "d5be2fc7-9300-42ac-b424-67400d5428fa"
        }
      ],
      "entitlements": [
        {
          "value": "workspace-access"
        },
        {
          "value": "allow-cluster-create"
        }
      ],
      "groups": [
        {
          "$ref": "Groups/469756445755404",
          "display": "gadp-bpi-externals",
          "type": "indirect",
          "value": "469756445755404"
        },
        {
          "$ref": "Groups/463224550509226",
          "display": "stla-bi",
          "type": "indirect",
          "value": "463224550509226"
        },
        {
          "$ref": "Groups/15627184289678",
          "display": "na-ada-pii",
          "type": "indirect",
          "value": "15627184289678"
        },
        {
          "$ref": "Groups/467055250863313",
          "display": "gadp-bpi",
          "type": "direct",
          "value": "467055250863313"
        },
        {
          "$ref": "Groups/838767098601544",
          "display": "na-vin-pii",
          "type": "indirect",
          "value": "838767098601544"
        },
        {
          "$ref": "Groups/1061922922679577",
          "display": "gadp-bpi-power-users",
          "type": "direct",
          "value": "1061922922679577"
        },
        {
          "$ref": "Groups/71500950656921",
          "display": "gadp-all-users-local",
          "type": "indirect",
          "value": "71500950656921"
        },
        {
          "$ref": "Groups/418923804214672",
          "display": "na-prod-non-pii-data-reader-role",
          "type": "indirect",
          "value": "418923804214672"
        },
        {
          "$ref": "Groups/231578680053125",
          "display": "na-ada-pii-local",
          "type": "indirect",
          "value": "231578680053125"
        }
      ],
      "id": "2117923742015926",
      "name": {
        "givenName": "gadp-bpi-cdm-dbx-sp-na"
      },
      "roles": [
        {
          "value": "arn:aws:iam::338911072615:instance-profile/databricks-clusterlogs-gadp-bpi-us-east-1-prod-inst-profile"
        },
        {
          "value": "arn:aws:iam::338911072615:instance-profile/databricks-clusterlogs-gadp-dsci-kg-us-east-1-prod-inst-profile"
        },
        {
          "value": "arn:aws:iam::338911072615:instance-profile/databricks-clusterlogs-us-east-1-prod-inst-profile"
        }
      ],
      "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
      ],
      "short_name": "d5be2fc7_9300_42ac_b424_67400d5428fa",
      "userName": "d5be2fc7-9300-42ac-b424-67400d5428fa"
    },
    "root_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na",
    "file_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files",
    "artifact_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/artifacts",
    "state_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/state"
  },
  "resources": {
    "jobs": {
      "cdm_na_tiger_roads": {
        "deployment": {
          "kind": "BUNDLE",
          "metadata_file_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/state/metadata.json"
        },
        "edit_mode": "UI_LOCKED",
        "email_notifications": {
          "on_failure": [
            "heather.goldsby@stellantis.com"
          ]
        },
        "format": "MULTI_TASK",
        "job_clusters": [
          {
            "job_cluster_key": "Job_cluster",
            "new_cluster": {
              "autoscale": {
                "max_workers": 7,
                "min_workers": 2
              },
              "aws_attributes": {
                "first_on_demand": 1
              },
              "custom_tags": {
                "stla_environment": "dev"
              },
              "init_scripts": [
                {
                  "workspace": {
                    "destination": "/Workspace/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/util/gdal/mosaic-gdal-init.sh"
                  }
                }
              ],
              "node_type_id": "i3.4xlarge",
              "policy_id": "0004D5D665AAB8E2",
              "runtime_engine": "PHOTON",
              "spark_version": "13.3.x-scala2.12"
            }
          }
        ],
        "max_concurrent_runs": 4,
        "name": "[dev d5be2fc7_9300_42ac_b424_67400d5428fa] cdm_na_tiger_roads",
        "parameters": [
          {
            "default": "gadp_scratch",
            "name": "output_catalog"
          },
          {
            "default": "cdm_mapping",
            "name": "output_schema"
          }
        ],
        "permissions": [
          {
            "level": "CAN_MANAGE",
            "group_name": "gadp-ds"
          },
          {
            "level": "CAN_MANAGE",
            "group_name": "gadp-bpi-power-users"
          }
        ],
        "queue": {
          "enabled": true
        },
        "tags": {
          "dev": "d5be2fc7_9300_42ac_b424_67400d5428fa"
        },
        "tasks": [
          {
            "job_cluster_key": "Job_cluster",
            "libraries": [
              {
                "pypi": {
                  "package": "databricks-mosaic"
                }
              },
              {
                "pypi": {
                  "package": "apache-sedona"
                }
              },
              {
                "pypi": {
                  "package": "numpy\u003e=1.18.5,\u003c1.25.0"
                }
              },
              {
                "maven": {
                  "coordinates": "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1"
                }
              },
              {
                "maven": {
                  "coordinates": "org.datasyslab:geotools-wrapper:geotools-24.0"
                }
              }
            ],
            "notebook_task": {
              "base_parameters": {
                "index_resolution": "10",
                "taskName": "{{job.name}}",
                "zip_url": "https://www2.census.gov/geo/tiger/GENZ2022/shp/cb_2022_us_state_20m.zip"
              },
              "notebook_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/na_us_tiger/01_geospatial_index_us_states"
            },
            "task_key": "01_geospatial_index_us_states"
          },
          {
            "depends_on": [
              {
                "task_key": "01_geospatial_index_us_states"
              }
            ],
            "job_cluster_key": "Job_cluster",
            "libraries": [
              {
                "pypi": {
                  "package": "databricks-mosaic"
                }
              },
              {
                "pypi": {
                  "package": "apache-sedona"
                }
              },
              {
                "pypi": {
                  "package": "numpy\u003e=1.18.5,\u003c1.25.0"
                }
              },
              {
                "maven": {
                  "coordinates": "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1"
                }
              },
              {
                "maven": {
                  "coordinates": "org.datasyslab:geotools-wrapper:geotools-24.0"
                }
              }
            ],
            "notebook_task": {
              "base_parameters": {
                "index_resolution": "10",
                "taskName": "{{job.name}}",
                "zip_url": "https://www2.census.gov/geo/tiger/TIGER2022/ZCTA520/tl_2022_us_zcta520.zip"
              },
              "notebook_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/na_us_tiger/01_geospatial_index_zipcodes"
            },
            "task_key": "01_geospatial_index_zipcodes"
          },
          {
            "depends_on": [
              {
                "task_key": "01_geospatial_index_zipcodes"
              }
            ],
            "job_cluster_key": "Job_cluster",
            "libraries": [
              {
                "pypi": {
                  "package": "databricks-mosaic"
                }
              },
              {
                "pypi": {
                  "package": "apache-sedona"
                }
              },
              {
                "pypi": {
                  "package": "numpy\u003e=1.18.5,\u003c1.25.0"
                }
              },
              {
                "maven": {
                  "coordinates": "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1"
                }
              },
              {
                "maven": {
                  "coordinates": "org.datasyslab:geotools-wrapper:geotools-24.0"
                }
              }
            ],
            "notebook_task": {
              "base_parameters": {
                "index_resolution": "10",
                "roads_url": "https://www2.census.gov/geo/tiger/TIGER2023/ROADS/",
                "taskName": "{{job.name}}"
              },
              "notebook_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/na_us_tiger/02_geospatial_index_roads"
            },
            "task_key": "02_geospatial_index_roads"
          },
          {
            "depends_on": [
              {
                "task_key": "02_geospatial_index_roads"
              }
            ],
            "job_cluster_key": "Job_cluster",
            "libraries": [
              {
                "pypi": {
                  "package": "databricks-mosaic"
                }
              },
              {
                "pypi": {
                  "package": "apache-sedona"
                }
              },
              {
                "pypi": {
                  "package": "numpy\u003e=1.18.5,\u003c1.25.0"
                }
              },
              {
                "maven": {
                  "coordinates": "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1"
                }
              },
              {
                "maven": {
                  "coordinates": "org.datasyslab:geotools-wrapper:geotools-24.0"
                }
              }
            ],
            "notebook_task": {
              "base_parameters": {
                "pbf_path": "https://download.geofabrik.de/north-america-latest.osm.pbf",
                "taskName": "{{job.name}}"
              },
              "notebook_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/util/osm_download"
            },
            "task_key": "osm_download"
          },
          {
            "depends_on": [
              {
                "task_key": "osm_download"
              }
            ],
            "job_cluster_key": "Job_cluster",
            "libraries": [
              {
                "pypi": {
                  "package": "databricks-mosaic"
                }
              },
              {
                "pypi": {
                  "package": "apache-sedona"
                }
              },
              {
                "pypi": {
                  "package": "numpy\u003e=1.18.5,\u003c1.25.0"
                }
              },
              {
                "maven": {
                  "coordinates": "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1"
                }
              },
              {
                "maven": {
                  "coordinates": "org.datasyslab:geotools-wrapper:geotools-24.0"
                }
              }
            ],
            "notebook_task": {
              "base_parameters": {
                "taskName": "{{job.name}}"
              },
              "notebook_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/na_osm/01_geospatial_index_roads"
            },
            "task_key": "01_geospatial_index_roads_osm"
          },
          {
            "depends_on": [
              {
                "task_key": "01_geospatial_index_roads_osm"
              }
            ],
            "job_cluster_key": "Job_cluster",
            "libraries": [
              {
                "pypi": {
                  "package": "databricks-mosaic"
                }
              },
              {
                "pypi": {
                  "package": "apache-sedona"
                }
              },
              {
                "pypi": {
                  "package": "numpy\u003e=1.18.5,\u003c1.25.0"
                }
              },
              {
                "maven": {
                  "coordinates": "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1"
                }
              },
              {
                "maven": {
                  "coordinates": "org.datasyslab:geotools-wrapper:geotools-24.0"
                }
              }
            ],
            "notebook_task": {
              "base_parameters": {
                "index_resolution": "10",
                "taskName": "{{job.name}}"
              },
              "notebook_path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/files/src/na_us_tiger/03_speed_limit_data_for_roads"
            },
            "task_key": "03_speed_limit_data_for_roads"
          }
        ]
      }
    }
  },
  "sync": {}
}
pietern commented 5 months ago

@Ribeiro-Breno Thanks for reporting. We'll take a look shortly.

Where does the error surface on databricks bundle run? Do you have a stack trace?

I suspect the issue related to ebs_volume_iops pops up because a deploy with the newer CLI version will have used an updated Terraform provider version.

Ribeiro-Breno commented 5 months ago

@pietern yes, the error is with databricks bundle run. Follow the stack trace:

[debug]Evaluating: secrets.GADP_BPI_CDM_DBX_SP_SECRET_NA_PROD

[debug]Evaluating Index:

[debug]..Evaluating secrets:

[debug]..=> Object

[debug]..Evaluating String:

[debug]..=> 'GADP_BPI_CDM_DBX_SP_SECRET_NA_PROD'

[debug]=> '***'

[debug]Result: '***'

[debug]Evaluating condition for step: 'Run databricks bundle run cdm_na_tiger_roads --refresh-all'

[debug]Evaluating: success()

[debug]Evaluating success:

[debug]=> true

[debug]Result: true

[debug]Starting: Run databricks bundle run cdm_na_tiger_roads --refresh-all

[debug]Loading inputs

[debug]Loading env

Run databricks bundle run cdm_na_tiger_roads --refresh-all

[debug]/usr/bin/bash -e /home/runner/work/_temp/f56b6cad-7fac-4b27-95b4-8927b8c57814.sh

Error: exit status 1 Failed to marshal state to json: unsupported attribute "ebs_volume_iops" Error: Process completed with exit code 1.

[debug]Finishing: Run databricks bundle run cdm_na_tiger_roads --refresh-all

About the "panic error", shouldn't be fixed into v 0.220.0? Follow the "panic error" stack trace using v 0.220.0:

[debug]Evaluating: secrets.GADP_BPI_CDM_DBX_SP_SECRET_NA_PROD

[debug]Evaluating Index:

[debug]..Evaluating secrets:

[debug]..=> Object

[debug]..Evaluating String:

[debug]..=> 'GADP_BPI_CDM_DBX_SP_SECRET_NA_PROD'

[debug]=> '***'

[debug]Result: '***'

[debug]Evaluating condition for step: 'Run databricks bundle run cdm_na_tiger_roads --refresh-all'

[debug]Evaluating: success()

[debug]Evaluating success:

[debug]=> true

[debug]Result: true

[debug]Starting: Run databricks bundle run cdm_na_tiger_roads --refresh-all

[debug]Loading inputs

[debug]Loading env

Run databricks bundle run cdm_na_tiger_roads --refresh-all

[debug]/usr/bin/bash -e /home/runner/work/_temp/3a32a2fd-c2d4-4ffc-93cb-030149dce250.sh

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0xe0fb4c]

goroutine 1 [running]: github.com/databricks/cli/bundle/run.(jobRunner).convertPythonParams(0xc0004e0b70?, 0xf?) github.com/databricks/cli/bundle/run/job.go:248 +0x4c github.com/databricks/cli/bundle/run.(jobRunner).Run(0xc0005dcde0, {0x154cc20, 0xc0003ed170}, 0xc0003a80e0) github.com/databricks/cli/bundle/run/job.go:158 +0x9a github.com/databricks/cli/cmd/bundle.newRunCommand.func1(0xc0003f1500, {0xc0003807e0, 0x1, 0x2}) github.com/databricks/cli/cmd/bundle/run.go:108 +0x5b1 github.com/spf13/cobra.(Command).execute(0xc0003f1500, {0xc0003807c0, 0x2, 0x2}) github.com/spf13/cobra@v1.8.0/command.go:983 +0xabc github.com/spf13/cobra.(Command).ExecuteC(0xc000005800) github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff github.com/spf13/cobra.(*Command).ExecuteContextC(...) github.com/spf13/cobra@v1.8.0/command.go:1048 github.com/databricks/cli/cmd/root.Execute(0x154cbe8?) github.com/databricks/cli/cmd/root/root.go:99 +0x5b main.main() github.com/databricks/cli/main.go:11 +0x2a Error: Process completed with exit code 2.

[debug]Finishing: Run databricks bundle run cdm_na_tiger_roads --refresh-all

{ "ProjectName":"cli", "Version":"0.220.0", "Branch":"HEAD", "Tag":"v0.220.0", "ShortCommit":"46f6cbcf", "FullCommit":"46f6cbcfc37d6dac836b027f5d60d824d7fdd16e", "CommitTime":"2024-05-22T09:08:27Z", "Summary":"v0.220.0", "Major":0, "Minor":220, "Patch":0, "Prerelease":"", "IsSnapshot":false, "BuildTime":"2024-05-22T09:14:38Z" }

pietern commented 5 months ago

For the first one, it happens if you databricks bundle run with v0.218.1, after having done a deploy with v0.220.0, correct?

Ribeiro-Breno commented 5 months ago

I didn't check deploing using v0.218.1 also. Let me try...

pietern commented 5 months ago

Is the source repository with bundle configuration checked out when running databricks bundle run?

When I look at the stack and why the panic happens, it seems like the configuration isn't loaded at all.

pietern commented 5 months ago

Could you also provide the output of databricks bundle run --debug when it produces the panic?

Ribeiro-Breno commented 5 months ago

@pietern when I deploy and run using v0218.1 it runs. I will provide you the debug for panic error using v0.220.0

pietern commented 5 months ago

Thank you!

Ribeiro-Breno commented 5 months ago 
13:11:38  INFO start pid=1567 version=0.220.0 args="databricks, bundle, run, cdm_na_tiger_roads, --refresh-all, --debug"
13:11:38 DEBUG Found bundle root at /home/runner/work/cdm_dsci_adas_geospatial/cdm_dsci_adas_geospatial (file /home/runner/work/cdm_dsci_adas_geospatial/cdm_dsci_adas_geospatial/databricks.yaml) pid=1567
13:11:38 DEBUG Apply pid=1567 mutator=load
13:11:38  INFO Phase: load pid=1567 mutator=load
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=EntryPoint
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=scripts.preinit
13:11:38 DEBUG No script defined for preinit, skipping pid=1567 mutator=load mutator=seq mutator=scripts.preinit
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=ProcessRootIncludes
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(jobs/job.yml)
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=VerifyCliVersion
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=EnvironmentsToTargets
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=InitializeVariables
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=LoadGitDetails
13:11:38 DEBUG Apply pid=1567 mutator=load mutator=seq mutator=SelectTarget(staging_na)
13:11:38 DEBUG Apply pid=1567 mutator=<func>
13:11:38 DEBUG Apply pid=1567 mutator=seq
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize
13:11:38  INFO Phase: initialize pid=1567 mutator=seq mutator=initialize
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=RewriteSyncPaths
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=MergeJobClusters
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=MergeJobTasks
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=MergePipelineClusters
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
13:11:38 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
13:11:38 DEBUG GET /oidc/.well-known/oauth-authorization-server
< HTTP/2.0 200 OK
< {
<   "authorization_endpoint": "https://dbc-deebf286-07d0.cloud.databricks.com/oidc/v1/authorize",
<   "code_challenge_methods_supported": [
<     "S256"
<   ],
<   "grant_types_supported": [
<     "client_credentials",
<     "authorization_code",
<     "refresh_token"
<   ],
<   "id_token_signing_alg_values_supported": [
13:11:39 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=FilterCurrentUserFromPermissions
13:11:39 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotateJobs
13:11:39 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotatePipelines
13:11:39 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
13:11:39 DEBUG DATABRICKS_TF_EXEC_PATH is not defined pid=1567 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
13:11:40 DEBUG Using Terraform at /home/runner/work/cdm_dsci_adas_geospatial/cdm_dsci_adas_geospatial/.databricks/bundle/staging_na/bin/terraform pid=1567 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
13:11:40 DEBUG DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=1567 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
13:11:40 DEBUG Environment variables for Terraform: HOME, PATH, DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET, DATABRICKS_AUTH_TYPE pid=1567 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
13:11:40 DEBUG Apply pid=1567 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
13:11:40 DEBUG No script defined for postinit, skipping pid=1567 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
13:11:40 DEBUG Apply pid=1567 mutator=seq mutator=terraform.Interpolate
13:11:40 DEBUG Apply pid=1567 mutator=seq mutator=terraform.Write
13:11:40 DEBUG Apply pid=1567 mutator=seq mutator=terraform:state-pull
13:11:40  INFO Opening remote state file pid=1567 mutator=seq mutator=terraform:state-pull
13:11:40 DEBUG GET /api/2.0/workspace/get-status?path=/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/state/terraform.tfstate
< HTTP/2.0 200 OK
< {
<   "created_at": 1715954313051,
<   "modified_at": 1716383484895,
<   "object_id": 890819685270758,
<   "object_type": "FILE",
<   "path": "/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/state/te... (15 more bytes)",
<   "resource_id": "890819685270758"
< } pid=1567 mutator=seq mutator=terraform:state-pull sdk=true
13:11:40 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/d5be2fc7-9300-42ac-b424-67400d5428fa/.bundle/cdm_dsci_adas_geospatial/staging_na/state/terraform.tfstate
< HTTP/2.0 200 OK
< <Streaming response> pid=1567 mutator=seq mutator=terraform:state-pull sdk=true
13:11:40  INFO Writing remote state file to local cache directory pid=1567 mutator=seq mutator=terraform:state-pull
13:11:40 DEBUG Apply pid=1567 mutator=seq mutator=terraform.Load
13:11:42  WARN unable to convert dynamic configuration to typed configuration: job cdm_na_tiger_roads is not defined pid=1567 mutator=seq
13:11:42 ERROR exit error: job cdm_na_tiger_roads is not defined pid=1567 mutator=seq
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0xe0fb4c]

goroutine 1 [running]:
github.com/databricks/cli/bundle/run.(*jobRunner).convertPythonParams(0xc0004e1230?, 0xf?)
    github.com/databricks/cli/bundle/run/job.go:248 +0x4c
github.com/databricks/cli/bundle/run.(*jobRunner).Run(0xc0004559c0, {0x154cc20, 0xc000543230}, 0xc0000f1180)
    github.com/databricks/cli/bundle/run/job.go:158 +0x9a
github.com/databricks/cli/cmd/bundle.newRunCommand.func1(0xc000541b00, {0xc000543050, 0x1, 0x3})
    github.com/databricks/cli/cmd/bundle/run.go:[10](https://github.com/StellantisGADP/cdm_dsci_adas_geospatial/actions/runs/9191961138/job/25279518838#step:4:11)8 +0x5b1
github.com/spf13/cobra.(*Command).execute(0xc000541b00, {0xc000542ff0, 0x3, 0x3})
    github.com/spf13/cobra@v1.8.0/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc000005800)
    github.com/spf13/cobra@v1.8.0/command.go:[11](https://github.com/StellantisGADP/cdm_dsci_adas_geospatial/actions/runs/9191961138/job/25279518838#step:4:12)15 +0x3ff
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
    github.com/spf13/cobra@v1.8.0/command.go:1048
github.com/databricks/cli/cmd/root.Execute(0x154cbe8?)
    github.com/databricks/cli/cmd/root/root.go:99 +0x5b
main.main()
    github.com/databricks/cli/main.go:11 +0x2a
Error: Process completed with exit code 2.
andrewnester commented 3 months ago

@Ribeiro-Breno could you please check if the error persists on the latest CLI version (0.224.1) ? Thank you!

Ribeiro-Breno commented 3 months ago

@andrewnester a solution for this error was found in issue https://github.com/databricks/cli/issues/1497