There is currently a permission issue for shared mode/user isolation for job clusters while using Brickflow. Per Databricks, local_disk0/.ephemeral_nfs is a directory used internally by the system, and users typically don't have direct access to it in shared cluster mode.
Important note:
Spark cannot directly interact with workspace files on compute configured with shared access mode, the purpose of this cluster is being secured, the action the import is trying to do is not allowed.
Describe the solution you'd like
Job cluster successfully working with Shared mode/User isolation mode in Brickflow.
Describe alternatives you've considered
We gave SELECT on all referenced tables and views in Single User mode to avoid Databricks limitation. This incurs tech debt as it means any new table/view in the upstream source can break our data pipeline .
There is currently a permission issue for shared mode/user isolation for job clusters while using Brickflow. Per Databricks, local_disk0/.ephemeral_nfs is a directory used internally by the system, and users typically don't have direct access to it in shared cluster mode.
Important note:
Spark cannot directly interact with workspace files on compute configured with shared access mode, the purpose of this cluster is being secured, the action the import is trying to do is not allowed.
Reference: https://docs.databricks.com/en/files/workspace-interact.html#read-data-workspace-files
Cloud Information
Describe the solution you'd like Job cluster successfully working with Shared mode/User isolation mode in Brickflow.
Describe alternatives you've considered We gave SELECT on all referenced tables and views in Single User mode to avoid Databricks limitation. This incurs tech debt as it means any new table/view in the upstream source can break our data pipeline .
Additional context None