databrickslabs / dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines
Other
125 stars 54 forks source link

Not able to onboard and deploy in interactive python terminal #35

Open mmicikevicius opened 4 months ago

mmicikevicius commented 4 months ago

Hello,

I'm trying to onboard and deploy DLT-META with all the default parameters in terminal, but getting this error:

Traceback (most recent call last): File "/Users/mantasmicikevicius/dlt-meta/src/cli.py", line 500, in main(sys.argv[1:]) File "/Users/mantasmicikevicius/dlt-meta/src/cli.py", line 496, in main MAPPINGcommand File "/Users/mantasmicikevicius/dlt-meta/src/cli.py", line 467, in onboard dltmeta.onboard(cmd) File "/Users/mantasmicikevicius/dlt-meta/src/cli.py", line 158, in onboard self._ws.dbfs.upload(cmd.dbfs_path + f"/dltmeta_conf/{onboarding_filename}", ob_file, overwrite=True) File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/mixins/files.py", line 320, in upload with self.open(path, write=True, overwrite=overwrite) as dst: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/mixins/files.py", line 316, in open return _DbfsIO(self, path, read=read, write=write, overwrite=overwrite) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/mixins/files.py", line 38, in init elif write: self._created = api.create(path, overwrite=overwrite) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/service/files.py", line 382, in create res = self._api.do('POST', '/api/2.0/dbfs/create', body=body, headers=headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/core.py", line 128, in do return retryable(self._perform)(method, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/retries.py", line 54, in wrapper raise err File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/retries.py", line 33, in wrapper return func(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/mantasmicikevicius/dlt-meta/.venv/lib/python3.11/site-packages/databricks/sdk/core.py", line 221, in _perform raise self._make_nicer_error(response=response, payload) from None databricks.sdk.errors.platform.ResourceAlreadyExists: A file or directory already exists at the input path dbfs:/dlt-meta_cli_demo/dltmeta_conf/onboarding.json. Error: exit status 1

ravi-databricks commented 4 months ago

@mmicikevicius , did you follow steps given here? Above error says onboarding file already exists, Can you details about options to onboard cli?