search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.42k stars 643 forks source link

Fix path substitution for file:// URIs #1251

Closed nfzd closed 1 month ago

nfzd commented 2 months ago

Related Issue \ discussion

Fix for #1233. See also #1217.

Patch Description

Perform url canonization in StorageHelper.get_driver_direct_access(). It is performed in all relevant methods on StorageHelper: here and here and here and here and here and here and here.

Testing Instructions

Described in #1233.

Other Information

Test for dataset creation and loading. Loading works as expected. Creation works and will write the data to the path with substitution applied while the stored name will not have substitution applied (this is the expected behavior).

I could only test for file://. Could not check the effect on other uri types (e.g., AWS).

nfzd commented 1 month ago

Is there anything missing / any way to speed this up?