Delta Write Fused Mount

[bill-ash]

Have been really enjoying your experiments. Thanks for sharing the notebooks!

Tried replicating writing to delta but keep running into this error even with storage_options:

OSError: Generic LocalFileSystem error: Unable to rename file: No such file or directory (os error 2)

Is there a specific combo of deltalake, and pyarrow that is needed?

Trying with the below:

import os 
import duckdb
import pandas as pd 
from deltalake import write_deltalake

df = pd.DataFrame({"one": [1, 2, 3]})
token = notebookutils.credentials.getToken("storage")

os.environ['MOUNT_ALLOW_UNSAFE_RENAME'] = "true"

storage_options = {
    "allow_unsafe_rename": "true",
    "bearer_token": token,
    }

# This syntax does not work... 
# url = "/lakehouse/default/Tables/test/newtable"
url = "abfss://{space}@onelake.dfs.fabric.microsoft.com/testing.Lakehouse/Tables/test/newtable/"
write_deltalake(url, df, mode="overwrite", storage_options=storage_options)

I can however query with duckdb using:

duckdb.sql("select * from delta_scan('/lakehouse/default/Tables/test/newtable')")

Any feedback would be appreciated!

[djouallah]

sorry for that, you need to install deltalake 0.18.2 and add engine='rust' for the mount to work, it is a current limitation and hopefully with 0.19.3 it will be fixed ( 0.19.2 has the same bug)

write_deltalake(url, df, mode="overwrite",engine='rust', storage_options={"allow_unsafe_rename": "true" })

let me know if it fix your issue

https://github.com/delta-io/delta-rs/issues/2860

[bill-ash]

Thanks for the link! That did the trick.