Closed Bernolt closed 1 month ago
It might not be a bug from delta-rs perspective, however, it would be helpful to have some insights on the underlying file system operation performed.
Afaik, databricks volumes are fuse mounted, so this is not an bug. If you want to write to mounted storage that doesn't support CopyIfNotExists, you can pass this to the writer:
storage_options = {"allow_unsafe_rename": "true"}
Thanks, solved my issue.
Environment
Delta-rs version: 0.17.4
Binding: python (pyarrow engine)
Environment:
Bug
What happened: From a python application running on a Databricks cluster, I want to write to an append-only delta table. The function is called as follows:
However, I am getting the below error:
What you expected to happen: As Databricks supports copy/rename/delete operations, I would expect it to work. As far as I know Databricks use a Local File System API, which emulates a filesystem on top of a cloud storage.
How to reproduce it: I made the below notebook to reproduce the error. It needs to be run from a Databricks Runtime.
More details: