search

databricks / containers

Sample base images for Databricks Container Services
Apache License 2.0
167 stars 118 forks source link

Error accessing unity catalog volumes #138

Open tschutte opened 1 year ago

tschutte commented 1 year ago

I'm attempting to use a custom container which is capable of reading data from Unity Catalog volumes. Before I create my custom container I'm testing using the out of the box containers provided by Databricks. So far I'm unable to find a configuration that is able to successfully access volumes. Is this functionality not supported yet, or am I missing a setting required to let this work?

Right now I'm testing using the standard:13.3-LTS image, which appears to be the most comprehensive.

The error I'm receiving is "IllegalArgumentException: Cannot access the UC Volume path from this location. Path was ... ".

Are there any tips I should try next?

helenxl commented 1 year ago

Following. We have the same issue.
The error we have is

IllegalArgumentException: Cannot access the UC Volume path from
this location. Path was /Volumes/

I suspect we just need to map /Volumes to the path underlying UC Volumes. Can the maintainers share a snippet that does this?

Jesusmruiz1 commented 12 months ago

Hi....some updates here? I find the same problem

exai-sukh commented 12 months ago

Adding the following spark config helps with this issue:

spark.databricks.unityCatalog.volumes.enabled true