Open erickfigueiredo opened 1 month ago
Has anyone ever faced this problem?
I'm also getting this error lately: Generic MicrosoftAzure error: Error after 10 retries in 3.296507315s, max_retries:10, retry_timeout:180s, source:HTTP status server error (503 Service Unavailable) for url (https://onelake.blob.fabric.microsoft.com/xxxxxxxxxxxxxx/Tables/dddddddddddd/_delta_log/_last_checkpoint).
The path I am using is: abfss://ws-name>@onelake.dfs.fabric.microsoft.com/<lakehousename.Lakehouse/Tables/
this should work, using 0.18.2 and above
from azure.identity import ClientSecretCredential, AuthenticationRequiredError
credential = ClientSecretCredential(
client_id = "appId",
client_secret="secret",
tenant_id= "tenantId"
)
access_token = credential.get_token("https://storage.azure.com/.default").token
storage_options= {"bearer_token": access_token, "use_fabric_endpoint": "true"}
from deltalake import DeltaTable
scada = DeltaTable('abfss://workspace@onelake.dfs.fabric.microsoft.com/Lakehousename.Lakehouse/Tables/xxxxx',storage_options=storage_options)
Environment
Delta-rs version: 0.16.0
Environment:
Bug
What happened: I'm facing an issue when using the deltalake lib to save / loading data to Azure Blob Storage. Sometimes, I'm getting the following error:
What you expected to happen: I expected to load the data from the Delta table and convert it to a Pandas DataFrame without any errors.
How to reproduce it:
More details: I was looking for a parameter like max_retries but couldn't find anything related. Does anyone know a solution or workaround for this issue? I didn't find an approach in the docs: https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html