[FEATURE] better use of local disk for registering models

stikkireddy / mlflow-extensions

Deploy models quickly to databricks via mlflow based serving infra.

Apache License 2.0

19 stars 11 forks source link

Open stikkireddy opened 2 months ago

stikkireddy commented 2 months ago

Clusters typically come with ~140 gb and we want to download model to /local_disk0 which is usually very large for all gpu vms

check if /local_disk0 exists and check if it has a lot of available space and prioritize using that instead and then use home directory on the cluster
try to touch a test file in there to check if the user has permissions to use that directory.