dbx destroy does not delete artefacts from dbfs

databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.

https://dbx.readthedocs.io

Other

438 stars 120 forks source link

dbx destroy does not delete artefacts from dbfs #718

Open alanmazankiewicz opened 1 year ago

alanmazankiewicz commented 1 year ago

Expected Behavior

Directory at <artifact_location>/<hash> should be deleted / empty

Current Behavior

Directory at <artifact_location>/<hash> contains wheel, config files, ...

Steps to Reproduce (for bugs)

dbx init my-proj dbx deploy test-proj-sample-etl <Check exact path to artifacts for this deployment (hash value), e.g. in workflow definition over UI> dbx destroy test-proj-sample-etl>

Your Environment

dbx version used: 0.8.9

guillesd commented 1 year ago

I think this is a bit related to my issue https://github.com/databrickslabs/dbx/issues/713, dbx creates a directory everytime you do a deployment but I wonder how it can keep track of all of this... My simple answer is that it doesn't, so it does not know what to destroy.