mgyucht
commented
4 weeks ago The SDK's DBUtils implementation has some restrictions compared to DBR's. The SDK can use the Databricks public REST API to interact with cloud storage. Unfortunately, that API is limited compared to what is possible to access in a cluster. As an example: a cluster may have an IAM role associated with it and thus have access to a specific S3 bucket, but when running locally you may not have that access.
That said: support for /Volumes paths was just added to the SDK in release v0.27.0. We've also added a better error message to clarify when you have specified a scheme that is not supported in the SDK's DBUtils implementation. Please try this out and let us know if this addresses your problem.
Description I need do some file manipulations via dbutils. According to the Databricks Connect V2 docs here: https://docs.databricks.com/en/dev-tools/databricks-connect/python/databricks-utilities.html we should be able to use dbutils from this sdk. However, this does not seem to work at all for external volumes. In a Notebook I can easily use dbutils.fs commands on external volumes via the URI 's3://[bucket]/[prefix]' or via /Volumes/[catalog]/[schema]/[external_volume_name] dbfs location, however, when I use dbutils from this sdk when running code via Databricks Connect I get and error
databricks.sdk.core.DatabricksError: No operations allowed on this pathwhen attempting to access an external volume via /Volumes. Is this not possible at all? I can only seem to manipulate dbfs locations outside of /Volumes. How can I perform file system operations on other cloud storage locations? What's even more confusing when you pass an s3 URI to dbutils.fs.ls when running locally in python the SDK seems to ignore the s3://[bucket] part and just assumes that is referring to the root location '/' and returns the listing from there.