Databricks CLI v0.224.1 version errors for `databricks bundle deploy`

weatherlym663 commented 2 months ago

Describe the issue

Updated the Databricks CLI version from v0.218.1 to v0.224.1 and databricks bundle deploy command is failing with error Error: no files found for ...-py3-none-any.whl - no other changes made besides changing the databricks cli version

Downgrading to v0.218.1 and the deployment is successful as expected

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

Build the wheel (we're using pdm build)
See the build files in dist directory
Run databricks bundle deploy
See the error - this was working for the previous version of databricks CLI
The dist directory is now gone

Expected Behavior

Python wheel should be deployed to databricks

Actual Behavior

An error saying the build files can't be found, however they do exist

OS and CLI version

Mac 14.4.1 Python 3.11.0 PDM 2.17.1 Databricks CLI v0.224.1

Is this a regression?

Yes - works as expected for databricks cli version v0.218.1

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks clusters list --log-level=debug. Redact if needed

andrewnester commented 2 months ago

@weatherlym663 could you please provide your bundle configuration so we could see what might go wrong? Thanks!

HansjoergW commented 1 month ago

I have the same problem. Version 0.224.0 is working, and it fails from Version 0.224.1 onwards.

Error message is

Error: no files found for /builds/diemobiliar/swe/wdh/sitinfra/wdh-sitinfra-housekeeping-pylib/dist/lib/cachetools-5.4.0-py3-none-any.whl

Everything is working fine when switching back to version 0.224.0

There is one wheel file in the ./dist/ folder and multiple wheel files in the /dist/lib folder

resources/cleanup.yml

# The main job for clean_up_job.
resources:
  jobs:
    clean_up_job:
      name: clean_up_job

      schedule:
        # Run every day at 8:37 AM
        quartz_cron_expression: '26 17 3 * * ?'
        timezone_id: Europe/Amsterdam

      tasks:
        - task_key: housekeeping         
          job_cluster_key: job_cluster
          python_wheel_task:
            package_name: had_sitinfra_housekeeping
            entry_point: housekeep
          libraries:
            - whl: ../dist/*.whl
            - whl: ../dist/lib/cachetools-5.4.0-py3-none-any.whl
            - whl: ../dist/lib/certifi-2024.7.4-py3-none-any.whl
            - whl: ../dist/lib/charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
            - whl: ../dist/lib/databricks_sdk-0.29.0-py3-none-any.whl
            - whl: ../dist/lib/google_auth-2.32.0-py2.py3-none-any.whl
            - whl: ../dist/lib/idna-3.7-py3-none-any.whl
            - whl: ../dist/lib/pyasn1_modules-0.4.0-py3-none-any.whl
            - whl: ../dist/lib/pyasn1-0.6.0-py2.py3-none-any.whl
            - whl: ../dist/lib/requests-2.32.3-py3-none-any.whl
            - whl: ../dist/lib/rsa-4.9-py3-none-any.whl
            - whl: ../dist/lib/urllib3-2.2.2-py3-none-any.whl
    ...

databricks.yml

bundle:
  name: sitinfra_housekeeping

include:
  - resources/*.yml

targets:
  # The 'dev' target, for development purposes. This target is the default.
  dev:
    # We use 'mode: development' to indicate this is a personal development copy:
    # - Deployed resources get prefixed with '[dev my_user_name]'
    # - Any job schedules and triggers are paused by default
    # - The 'development' mode is used for Delta Live Tables pipelines
    mode: development
    default: true
    workspace:
      # deployment is also to the sit-infra runtime
      host: https://adb-xxxx.15.azuredatabricks.net/
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}
    run_as:
      # This runs as the user which deploys it
      # see https://docs.databricks.com/dev-tools/bundles/permissions.html.
      user_name: ${workspace.current_user.userName}

  # The 'sit' target, used for "production" deployment.
  sit:
    # We use 'mode: production' to indicate this is a production deployment.
    # Doing so enables strict verification of the settings below.
    mode: production
    workspace:
      # deployment is to the sit-infra runtime
      host: https://adb-xxxx.15.azuredatabricks.net/
    run_as:
      # This runs as a service principal dlp-gitlabrunner-iac-wdh-sit-mi
      service_principal_name: "xxxxx"

andrewnester commented 1 month ago

Thanks for the detailed response! It's indeed an issue on our side and the fix will be released in the upcoming CLI release

databricks / cli