No such file or directory: '/demo' regardless of input arguments for databricks labs dlt-meta onboard

[brojoe-1]

Hi, when running dlt-meta onboard, I was stuck on issue https://github.com/databrickslabs/dlt-meta/issues/51 that is not yet released, so I switched to your branch 51-dlt-meta-cli-onboard-command-issue-for-azure-databrickssdkerrorsplatformresourcealreadyexists

However, there are more issues there - regardless of input arguments for dlt-meta onboard, it tries to read from '/demo' that does not exist. Since I am on a mac, it is not possible for me to move the demo folder to root https://support.apple.com/en-gb/101400

I would assume that the 'dlt-meta onboard cmd' would try to read from folder specified in 'Provide onboarding files local directory', but that is not the case.

See my use of cmd and stacktrace:

sudo databricks labs dlt-meta onboard --debug --profile=test
15:50:54 INFO start pid=94725 version=0.223.1 args="databricks, labs, dlt-meta, onboard, --debug, --profile=test" 15:50:54 INFO Overriding login profile: test pid=94725 15:50:54 DEBUG Loading test profile from /Users/dev/.databrickscfg pid=94725 sdk=true 15:50:54 DEBUG Passing down environment variables: DATABRICKS_HOST, DATABRICKS_TOKEN pid=94725 15:50:54 DEBUG Forwarding subprocess: /Users/dev/.databricks/labs/dlt-meta/state/venv/bin/python3 /Users/dev/dlt-meta/src/cli.py {"command":"onboard","flags":{"log_level":"debug"},"output_type":""} pid=94725 15:50:54 DEBUG starting: /Users/dev/.databricks/labs/dlt-meta/state/venv/bin/python3 /Users/dev/dlt-meta/src/cli.py {"command":"onboard","flags":{"log_level":"debug"},"output_type":""} pid=94725 Provide onboarding file path (default: demo/conf/onboarding.template): Provide onboarding files local directory (default: demo/): Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): dbfs:/dlt-meta_cli_demo_100 Provide databricks runtime version (default: 15.3.x-scala2.12): Run onboarding with unity catalog enabled? [0] False [1] True Enter a number between 0 and 1: 1 Provide unity catalog name: dlt_test Provide dlt meta schema name (default: dlt_meta_dataflowspecs_f5eda329176c4678bd2cf2c499c270e2): Provide dlt meta bronze layer schema name (default: dltmeta_bronze_2a2e2b8c465244e5a22743b4d0013eed): Provide dlt meta silver layer schema name (default: dltmeta_silver_481fbe8078ef4267aec8966d4efb9ec1): Provide dlt meta layer [0] bronze [1] bronze_silver [2] silver Enter a number between 0 and 2: 1 Provide bronze dataflow spec table name (default: bronze_dataflowspec): Provide silver dataflow spec table name (default: silver_dataflowspec): Overwrite dataflow spec? [0] False [1] True Enter a number between 0 and 1: 1 Provide dataflow spec version (default: v1): Provide environment name (default: prod): test Provide import author name (default: dev): test_auth Provide cloud provider name [0] aws [1] azure [2] gcp Enter a number between 0 and 2: 1 Update workspace/dbfs paths, unity catalog name, bronze/silver schema names in onboarding file? [0] False [1] True Enter a number between 0 and 1: 1 Traceback (most recent call last): File "/Users/dev/dlt-meta/src/cli.py", line 503, in main(*sys.argv[1:]) File "/Users/dev/dlt-meta/src/cli.py", line 499, in main MAPPINGcommand File "/Users/dev/dlt-meta/src/cli.py", line 470, in onboard dltmeta.onboard(cmd) File "/Users/dev/dlt-meta/src/cli.py", line 160, in onboard self._ws.dbfs.copy(cmd.onboarding_files_dir_path, File "/Users/dev/.databricks/labs/dlt-meta/state/venv/lib/python3.9/site-packages/databricks/sdk/mixins/files.py", line 603, in copy with child.open(read=True) as reader: File "/Users/dev/.databricks/labs/dlt-meta/state/venv/lib/python3.9/site-packages/databricks/sdk/mixins/files.py", line 351, in open return self._path.open(mode='wb' if overwrite else 'rb' if read else 'xb') File "/Users/dev/.pyenv/versions/3.9.6/lib/python3.9/pathlib.py", line 1242, in open return io.open(self, mode, buffering, encoding, errors, newline, File "/Users/dev/.pyenv/versions/3.9.6/lib/python3.9/pathlib.py", line 1110, in _opener return self._accessor.open(self, flags, mode) FileNotFoundError: [Errno 2] No such file or directory: '/demo' Error: exit status 1 15:52:15 ERROR failed execution pid=94725 exit_code=1 error="exit status 1"

thank you for any help.

[ravi-databricks]

@brojoe-1 , can you please test above error in branch: feature/v.0.0.8. Here are steps:

 git clone https://github.com/databrickslabs/dlt-meta.git 

cd dlt-meta

 git checkout feature/v0.0.8

 python -m venv .venv 

source .venv/bin/activate 

 pip install databricks-sdk 

dlt_meta_home=$(pwd)

export PYTHONPATH=$dlt_meta_home

databricks labs dlt-meta onboard --debug --profile=test

[ravi-databricks]

fixed in feature/v0.0.8