search

databricks / cli

Databricks CLI
Other
150 stars 56 forks source link

Relative path definition in "paths" mapping not working properly #1755

Open lachniej opened 2 months ago

lachniej commented 2 months ago

In attempting to use the paths mapping within the sync in a databricks.yml I am getting various errors when attempting to sync a common folder at the same level as the bundle directory (not the bundle root). It is producing errors both with pydabs and dabs default initialization templates.

Configuration

databricks cli version: 0.227.0 and 0.227.1 pydabs version for pydabs testing: 0.5.1 python version: 3.11

Folder Structure - dabs --using default dabs template-- 

├── common/
│   ├── src/
│   │  ├── __init__.py
│   │  ├── hello.py
└── my_bundle/
│   ├── databrick.yml
│   ├── src/
│   ├── dist/
│   │  ├── test_this-0.0.1+20240906.124245-py3-none-any.whl
│   ├── resources/
│   │  ├── my_bundle_job.yml
│   ├── ...

Folder Structure - pydabs --using default pydabs template--

├── common/
│   ├── src/
│   │  ├── __init__.py
│   │  ├── hello.py
└── my_bundle/
│   ├── databrick.yml
│   ├── src/
│   │  ├── my_bundle/ 

│   ├── ...

databricks.yml - dabs

# This is a Databricks asset bundle definition for test_this.
# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
bundle:
  name: test_this

include:
  - resources/*.yml

sync:
  paths:
    - ../common

targets:
  # The 'dev' target, for development purposes. This target is the default.
  dev:
    # We use 'mode: development' to indicate this is a personal development copy:
    # - Deployed resources get prefixed with '[dev my_user_name]'
    # - Any job schedules and triggers are paused by default
    # - The 'development' mode is used for Delta Live Tables pipelines
    mode: development
    default: true
    workspace:
      host: https://adb-xxx-azuredatabricks.net

  ## Optionally, there could be a 'staging' target here.
  ## (See Databricks docs on CI/CD at https://docs.databricks.com/dev-tools/bundles/ci-cd.html.)
  #
  # staging:
  #   workspace:
  #     host: https://adb-xxx-azuredatabricks.net

  # The 'prod' target, used for production deployment.
  prod:
    # We use 'mode: production' to indicate this is a production deployment.
    # Doing so enables strict verification of the settings below.
    mode: production
    workspace:
      host: https://adb-xxx-azuredatabricks.net
      # We always use /Users/xxx@.com for all resources to make sure we only have a single copy.
      # If this path results in an error, please make sure you have a recent version of the CLI installed.
      root_path: /Users/xxx@.com/.bundle/${bundle.name}/${bundle.target}
    run_as:
      # This runs as xxx@.com in production. We could also use a service principal here,
      # see https://docs.databricks.com/dev-tools/bundles/permissions.html.
      user_name: xxx@.com

resources/my_bundle_job.yml - dabs

resources:
  jobs:
    my_bundle_job:
      name: my_bundle_job

      schedule:
        # Run every day at 8:37 AM
        quartz_cron_expression: '44 37 8 * * ?'
        timezone_id: Europe/Amsterdam

      email_notifications:
        on_failure:
          - xxx@.com

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ../src/notebook.ipynb

        - task_key: refresh_pipeline
          depends_on:
            - task_key: notebook_task
          pipeline_task:
            pipeline_id: ${resources.pipelines.test_this_pipeline.id}

        - task_key: main_task
          depends_on:
            - task_key: refresh_pipeline
          job_cluster_key: job_cluster
          python_wheel_task:
            package_name: test_this
            entry_point: main
          libraries:
            # By default we just include the .whl file generated for the test_this package.
            # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
            # for more information on how to add other libraries.
            - whl: ../dist/*.whl

      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_D3_v2
            autoscale:
                min_workers: 1
                max_workers: 4

databricks.yml - pydabs

# This is a Databricks asset bundle definition for pydabs_5_1.
# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
bundle:
  name: pydabs_5_1
  databricks_cli_version: ">= 0.227.1"

include:
  - "resources/*.yml"

experimental:
  pydabs:
    enabled: true
    venv_path: .venv
    # list of Python packages to import to discover resources, resource generator
    # and mutators
    import:
    - "pydabs_5_1"
  scripts:
    preinit: |
      # create .venv if it doesn't exist
      if [ ! -e .venv ]; then
        python=$(command -v python3.10 || command -v python3.11 || command -v python3.12) || (
          echo "ERROR: Python 3.10 or higher is required to create virtual environment";
          exit 1;
        )

        $python -m venv .venv
        .venv/bin/pip install -r requirements-dev.txt
      fi

artifacts:
  default:
    type: whl
    path: .
    build: .venv/bin/python3 setup.py bdist_wheel

sync:
  exclude:
    - "src"
  paths:
    - ../common

variables:
  warehouse_id:
    description: Warehouse ID for SQL tasks
    default: change-me
  default_cluster_spec:
    description: Spec for 'Default' job cluster
    default:
      spark_version: 13.3.x-scala2.12
      node_type_id: Standard_D3_v2
      autoscale:
        min_workers: 1
        max_workers: 4

targets:
  # The 'dev' target, used for development purposes.
  # Whenever a developer deploys using 'dev', they get their own copy.
  dev:
    # We use 'mode: development' to make sure everything deployed to this target gets a prefix
    # like '[dev my_user_name]'. Setting this mode also disables any schedules and
    # automatic triggers for jobs and enables the 'development' mode for Delta Live Tables pipelines.
    mode: development
    default: true
    workspace:
      host: https://adb-xxx-azuredatabricks.net

  # Optionally, there could be a 'staging' target here.
  # (See Databricks docs on CI/CD at https://docs.databricks.com/dev-tools/bundles/index.html.)
  #
  # staging:
  #  workspace:
  #    host: https://adb-xxx-azuredatabricks.net

  # The 'prod' target, used for production deployment.
  prod:
    # For production deployments, we only have a single copy, so we override the
    # workspace.root_path default of
    # /Users/${workspace.current_user.userName}/.bundle/${bundle.target}/${bundle.name}
    # to a path that is not specific to the current user.
    mode: production
    workspace:
      host: https://adb-xxx-azuredatabricks.net
      root_path: /Shared/.bundle/prod/${bundle.name}
    run_as:
      # This runs as xxx@.com in production. Alternatively,
      # a service principal could be used here using service_principal_name
      # (see Databricks documentation).
      user_name: xxx@.com

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

databricks bundle deploy --target dev

Expected Behavior

Expected behavior would be deployment of bundle along with the shared package for use across multiple bundles within the same repo

Actual Behavior

error - dabs

Error: no files match pattern: my_bundl/dist/*.whl
  at resources.jobs.my_bundle_job.tasks[2].libraries[0].whl
  in resources/my_bundle_job.yml:39:15

error - pydabs

Error: build failed default, error: chdir /Users/xxx/git_repos/my_bundle/my_bundle/my_bundle: no such file or directory

OS and CLI version

tested this with databricks cli version: 0.227.0 and 0.227.1

Is this a regression?

does not work in neither version since feature release

Debug Logs

dabs

09:43:08  INFO start pid=57009 version=0.227.0 args="databricks, bundle, deploy, --target, dev, --log-level=debug"
09:43:08 DEBUG Found bundle root at /Users/xxx/git_repos/pydabs5_1/pydabs_5_1 (file /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/databricks.yml) pid=57009
09:43:08 DEBUG Apply pid=57009 mutator=load
09:43:08  INFO Phase: load pid=57009 mutator=load
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=EntryPoint
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=scripts.preinit
Executing 'preinit' script
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=ProcessRootIncludes
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=VerifyCliVersion
09:43:08 ERROR Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0 pid=57009 mutator=load mutator=seq mutator=VerifyCliVersion
09:43:08 ERROR Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0 pid=57009 mutator=load mutator=seq
09:43:08 ERROR Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0 pid=57009 mutator=load
Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0

09:43:08 ERROR failed execution pid=57009 exit_code=1 error=AlreadyPrinted

pydabs


09:44:45  INFO start pid=57142 version=0.227.0 args="databricks, bundle, deploy, --target, dev, --log-level, debug"
09:44:45 DEBUG Found bundle root at /Users/xxx/git_repos/test_this (file /Users/xxx/git_repos/test_this/databricks.yml) pid=57142
09:44:45 DEBUG Apply pid=57142 mutator=load
09:44:45  INFO Phase: load pid=57142 mutator=load
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=EntryPoint
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=scripts.preinit
09:44:45 DEBUG No script defined for preinit, skipping pid=57142 mutator=load mutator=seq mutator=scripts.preinit
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/test_this_job.yml)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/test_this_pipeline.yml)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=VerifyCliVersion
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=EnvironmentsToTargets
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=InitializeVariables
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=LoadGitDetails
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=PythonMutator(load)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=validate:unique_resource_keys
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=SelectTarget(dev)
09:44:45 DEBUG Apply pid=57142 mutator=<func>
09:44:45 DEBUG Apply pid=57142 mutator=<func>
09:44:45 DEBUG Apply pid=57142 mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize
09:44:45  INFO Phase: initialize pid=57142 mutator=seq mutator=initialize
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=RewriteSyncPaths
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SyncDefaultPath
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SyncInferRoot
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergeJobClusters
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergeJobParameters
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergeJobTasks
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergePipelineClusters
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
09:44:45 DEBUG Loading profile DEFAULT because of host match pid=57142
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
09:44:45 DEBUG Loading profile DEFAULT because of host match pid=57142
09:44:45  INFO Ignoring pat auth, because databricks-cli is preferred pid=57142 sdk=true
09:44:45  INFO Ignoring basic auth, because databricks-cli is preferred pid=57142 sdk=true
09:44:45  INFO Ignoring oauth-m2m auth, because databricks-cli is preferred pid=57142 sdk=true
09:44:45 DEBUG Running command: /opt/homebrew/Cellar/databricks@0.227.0/0.227.0/bin/databricks auth token --host https://adb-xxx-azuredatabricks.net pid=57142 sdk=true
09:44:46  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-06 10:44:46.860721 -0400 EDT pid=57142 sdk=true
09:44:46 DEBUG Using Databricks CLI authentication with Databricks OAuth tokens pid=57142 sdk=true
09:44:46 DEBUG Running command: /opt/homebrew/Cellar/databricks@0.227.0/0.227.0/bin/databricks auth token --host https://adb-xxx-azuredatabricks.net pid=57142 sdk=true
09:44:46  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-06 10:44:46.860721 -0400 EDT pid=57142 sdk=true
09:44:47 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "displayName": "xxx",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "xxx@.com"
<     }
<   ],
<   "externalId": "037a9f7f-ec55-423b-a30f-4734a2603845",
<   "groups": [
<     {
<       "$ref": "Groups/778794051400239",
<       "display": xxx,
<       "type": "direct",
<       "value": "778794051400239"
<     }
<   ],
<   "id": "4286441873933971",
<   "name": {
<     "familyName": "xxx",
<     "givenName": "xxx"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
<   ],
<   "userName": "xxx@.com"
< } pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SetVariables
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveResourceReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SetRunAs
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=OverrideCompute
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
09:44:47  INFO Development mode: disabling deployment lock since bundle.deployment.lock.enabled is not set to true pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ApplyPresets
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=DefaultQueueing
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ExpandPipelineGlobPaths
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ConfigureWSFS
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=TranslatePaths
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ApplyBundlePermissions
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=FilterCurrentUserFromPermissions
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotateJobs
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotatePipelines
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG DATABRICKS_TF_EXEC_PATH is not defined pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG Using Terraform at /Users/xxx/git_repos/test_this/.databricks/bundle/dev/bin/terraform pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG Environment variables for Terraform: DATABRICKS_AUTH_TYPE, HOME, PATH, TMPDIR, DATABRICKS_USER_AGENT_EXTRA, DATABRICKS_HOST pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
09:44:47 DEBUG No script defined for postinit, skipping pid=57142 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build
09:44:47  INFO Phase: build pid=57142 mutator=seq mutator=build
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
09:44:47 DEBUG No script defined for prebuild, skipping pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
09:44:47  INFO Detecting Python wheel project... pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
09:44:47  INFO Found Python wheel project at /Users/xxx/git_repos/test_this pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq mutator=artifacts.Infer(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq mutator=artifacts.Infer(test_this) mutator=artifacts.whl.Infer(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(test_this) mutator=artifacts.whl.Prepare(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq mutator=artifacts.whl.Build(test_this)
Building test_this...
09:44:47  INFO Build succeeded pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq mutator=artifacts.whl.Build(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq mutator=artifacts.ExpandGlobs(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.postbuild
09:44:47 DEBUG No script defined for postbuild, skipping pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.postbuild
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy
09:44:47  INFO Phase: deploy pid=57142 mutator=seq mutator=deploy
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.predeploy
09:44:47 DEBUG No script defined for predeploy, skipping pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.predeploy
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire
09:44:47  INFO Skipping; locking is disabled pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
09:44:49 DEBUG GET /api/2.0/workspace/get-status?path=/Users/xxx@.com/.bundle/test_this/dev/state/terraform.tfstate&return_export_info=true
< HTTP/2.0 200 OK
< {
*****
< } pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
09:44:50 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/xxx@.com/.bundle/test_this/dev/state/terraform.tfstate
< HTTP/2.0 200 OK
< <Streaming response> pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
09:44:50 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
09:44:50  INFO Opening remote deployment state file pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
09:44:51 DEBUG GET /api/2.0/workspace/get-status?path=/Users/xxx@.com/.bundle/test_this/dev/state/deployment.json&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1725583110251,
<   "modified_at": 1725626329939,
<   "object_id": 3405635552738227,
<   "object_type": "FILE",
<   "path": "/Users/xxx@.com/.bundle/test_this/dev/state/deployment.json",
<   "resource_id": "xxx"
< } pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull sdk=true
09:44:51 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/xxx@.com/.bundle/test_this/dev/state/deployment.json
< HTTP/2.0 200 OK
< <Streaming response> pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull sdk=true
09:44:51  INFO Local deployment state is the same or newer, ignoring remote state pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=ValidateGitDetails
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp
09:44:51 ERROR failed to delete /Users/xxx@.com/.bundle/test_this/dev/artifacts/.internal: unable to delete filer root pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp
09:44:51 DEBUG POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/xxx@.com/.bundle/test_this/dev/artifacts/.internal"
> }
< HTTP/2.0 200 OK
< {} pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp sdk=true
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=libraries.ExpandGlobReferences
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=libraries.ExpandGlobReferences
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
09:44:51  INFO Skipping; locking is disabled pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq
Error: no files match pattern: test_this/dist/*.whl
  at resources.jobs.test_this_job.tasks[2].libraries[0].whl
  in resources/test_this_job.yml:39:15

09:44:51 ERROR failed execution pid=57142 exit_code=1 error=AlreadyPrinted````
pietern commented 2 months ago

@lachniej Thanks for reporting the issue!

I didn't apply the relevant changes for library glob expansion as part of #1694 and that's where the first issue you list comes from. A fix for this issue is in #1756 and will be included in the next release later this week.

The PyDABs issue you mention (Error: build failed default, error: chdir) doesn't match the debug output.

Could you double check this?

lachniej commented 2 months ago

@pietern thanks for that info.

Yeah sorry, I forgot to switch the version back to the supported databricks cli version for pydabs 5.1. Here is actual error. 

10:24:54  INFO start pid=39810 version=0.228.0 args="databricks, bundle, deploy, --target, dev, --log-level=debug"
10:24:54 DEBUG Found bundle root at /Users/xxx/git_repos/pydabs5_1/pydabs_5_1 (file /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/databricks.yml) pid=39810
10:24:54 DEBUG Apply pid=39810 mutator=load
10:24:54  INFO Phase: load pid=39810 mutator=load
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=EntryPoint
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=scripts.preinit
Executing 'preinit' script
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=ProcessRootIncludes
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=VerifyCliVersion
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=EnvironmentsToTargets
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=InitializeVariables
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=LoadGitDetails
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG running: .venv/bin/python3 -m databricks.bundles.build --phase load --input /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2277372783/input.json --output /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2277372783/output.json --diagnostics /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2277372783/diagnostics.json pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.conditional_execution' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_conditional_execution_detect_anomalies pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.hello_world' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_hello_world_hello_world pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.library_dependencies' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_library_dependencies_cowsay_job pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.python_task' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.resource_generator' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered resource generator my_notebook_jobs pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.task_values' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_task_values_task_values pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.mutators' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered resource mutator default_job_cluster_mutator pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=validate:unique_resource_keys
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=SelectTarget(dev)
10:24:54 DEBUG Apply pid=39810 mutator=<func>
10:24:54 DEBUG Apply pid=39810 mutator=<func>
10:24:54 DEBUG Apply pid=39810 mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize
10:24:54  INFO Phase: initialize pid=39810 mutator=seq mutator=initialize
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=RewriteSyncPaths
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SyncDefaultPath
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SyncInferRoot
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergeJobClusters
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergeJobParameters
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergeJobTasks
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergePipelineClusters
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
10:24:54 DEBUG Loading profile DEFAULT because of host match pid=39810
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
10:24:54 DEBUG Loading profile DEFAULT because of host match pid=39810
10:24:54  INFO Ignoring pat auth, because databricks-cli is preferred pid=39810 sdk=true
10:24:54  INFO Ignoring basic auth, because databricks-cli is preferred pid=39810 sdk=true
10:24:54  INFO Ignoring oauth-m2m auth, because databricks-cli is preferred pid=39810 sdk=true
10:24:54 DEBUG Running command: /opt/homebrew/Cellar/databricks/0.228.0/bin/databricks auth token --host https://adb-xxx.azuredatabricks.net pid=39810 sdk=true
10:24:54  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-09 11:24:15.487025 -0400 EDT pid=39810 sdk=true
10:24:54 DEBUG Using Databricks CLI authentication with Databricks OAuth tokens pid=39810 sdk=true
10:24:54 DEBUG Running command: /opt/homebrew/Cellar/databricks/0.228.0/bin/databricks auth token --host https://adb-xxx.azuredatabricks.net pid=39810 sdk=true
10:24:54  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-09 11:24:15.487025 -0400 EDT pid=39810 sdk=true
10:24:55 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "displayName": "xxx",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "xxx"
<     }
<   ],
<   "externalId": "037a9f7f-ec55-423b-a30f-4734a2603845",
<   "groups": [
<     {
<       "$ref": "Groups/778794051400239",
<       "display": "xxx",
<       "type": "direct",
<       "value": "778794051400239"
<     }
<   ],
<   "id": "4286441873933971",
<   "name": {
<     "familyName": "xxx",
<     "givenName": "xxx"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
<   ],
<   "userName": "xxx"
< } pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SetVariables
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG running: .venv/bin/python3 -m databricks.bundles.build --phase init --input /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2355978857/input.json --output /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2355978857/output.json --diagnostics /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2355978857/diagnostics.json pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.conditional_execution' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_conditional_execution_detect_anomalies pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.hello_world' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_hello_world_hello_world pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.library_dependencies' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_library_dependencies_cowsay_job pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.python_task' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.resource_generator' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered resource generator my_notebook_jobs pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.task_values' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_task_values_task_values pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.mutators' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered resource mutator default_job_cluster_mutator pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_conditional_execution_detect_anomalies.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_conditional_execution_detect_anomalies.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_conditional_execution_detect_anomalies.tasks[2].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_hello_world_hello_world.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_hello_world_hello_world.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_library_dependencies_cowsay_job.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_library_dependencies_cowsay_job.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_task_values_task_values.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_task_values_task_values.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_task_values_task_values.tasks[1].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.notebook" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveResourceReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SetRunAs
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=OverrideCompute
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
10:24:55  INFO Development mode: disabling deployment lock since bundle.deployment.lock.enabled is not set to true pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ApplyPresets
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=DefaultQueueing
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ExpandPipelineGlobPaths
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ConfigureWSFS
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=TranslatePaths
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ApplyBundlePermissions
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=FilterCurrentUserFromPermissions
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotateJobs
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotatePipelines
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG DATABRICKS_TF_EXEC_PATH is not defined pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG Using Terraform at /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/.databricks/bundle/dev/bin/terraform pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG Environment variables for Terraform: DATABRICKS_USER_AGENT_EXTRA, DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, HOME, PATH, TMPDIR pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
10:24:55 DEBUG No script defined for postinit, skipping pid=39810 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build
10:24:55  INFO Phase: build pid=39810 mutator=seq mutator=build
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
10:24:55 DEBUG No script defined for prebuild, skipping pid=39810 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
10:24:55 DEBUG artifacts block is defined, skipping auto-detecting pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq mutator=artifacts.Infer(default)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(default)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(default) mutator=artifacts.whl.Prepare(default)
10:24:55 DEBUG open dir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: open /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory pid=39810
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq mutator=artifacts.whl.Build(default)
Building default...
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq mutator=artifacts.whl.Build(default)
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default)
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq
Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:
lachniej commented 2 months ago

@pietern not sure if you had resolved what is going on with pydabs, but with the pre-release cli version right now dabs is working but pydabs is still returning this

image
pietern commented 1 month ago

@lachniej Could you confirm if the latest version of the CLI resolves the issue?

It should work for the regular DABs case. I'm not sure about the PyDABs case.