Open nfzd opened 3 months ago
Does anyone have an idea what the problem could be or how to debug the issue?
Hi @nfzd ! Looks like the StorageHelper tries to access file://
links directly, without applying file substitution, and if the referenced file does not exist, then the program will raise an error.
We will need to fix this on our side (or if you wish to contribute you could open a PR that handles path substitutions in https://github.com/allegroai/clearml/blob/d4e136307cecd35b1e40174ad2b3132e981dc7c4/clearml/storage/helper.py#L1817).
The only workaround I can think of is forcing get_direct_access
to return None:
from clearml.storage.helper import _FileStorageDriver
_FileStorageDriver.get_direct_access = lambda *args: None
# should work
from clearml import Dataset
d = Dataset.get("d2412eff1f7f462fb6c81065e043cd8b")
Describe the bug
output_uri
.To reproduce
Contents of
clearml.conf
:In folder
/home/user/clearml_path_substitution
.Contents of file
create.py
:Contents of file
load.py
:Create dataset:
(Loading it at this point by running
load.py
works as expected.)Move the storage location:
Add the path substitution rule:
Try loading from the new location:
Expected behaviour
Loading should be possible from the new storage location using path substitution.
Environment