search

databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace
Other
230 stars 65 forks source link

Mount to DBFS #1271

Closed letssamu closed 2 months ago

letssamu commented 4 months ago

About mounts, in this documentation: https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage#deprecated-patterns-for-storing-and-accessing-data-from-azure-databricks

Says: "Databricks no longer recommends mounting external data locations to Databricks Filesystem."

And refers this documentation: https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts#--mount-adls-gen2-or-blob-storage-with-abfs

Says: "Mount points that use secrets are not automatically refreshed. If mounted storage relies on a secret that is rotated, expires, or is deleted, errors can occur, such as 401 Unauthorized. To resolve such an error, you must unmount and remount the storage."

Because of internal security policies, we have to create Client Secret frequently and we have to unmount and mount every time.

Is possible to evaluate other possibilities to not use mount for Cluster logs?

https://databrickslabs.github.io/overwatch/deployoverwatch/cloudinfra/azure//#configuring-overwatch-on-azure-databricks

Thank you.

mohanbaabu1996 commented 2 months ago

Hi @letssamu

Thanks for your suggestion, and apologize for the delay. As the platform supports only the mounted path in the cluster configuration, this has to be raised under cluster product team. I hope you understand.

image

Thanks!