search

tecosaur / DataToolkit.jl

Reproducible, flexible, and convenient data management
https://tecosaur.github.io/DataToolkit.jl
78 stars 4 forks source link

Specify storage location for downloaded datasets #29

Closed kescobo closed 8 months ago

kescobo commented 8 months ago

Not sure if this is the right repo for this, feel free to move it!

Occasionally, it would be useful to set the location of downloaded datasets, eg in an HPC environment, I want to be able to download to a fast scratch space, rather than a home folder that typically contains my projects. But I wouldn't want to hard-code that location into the Data.toml file, since that wouldn't be portable.

Right now, I typically do something like 

path = @load_preference("my_location"; default = get(ENV, "MY_LOCATION", "./my_location))

I think I can see how one could do this using the execution of arbitrary julia code in one of the plugins, but it seems like a common enough ask that it might be worth building it in.

tecosaur commented 8 months ago

So, if no particular path is hard-coded into the Data.toml, the default will be used. This default can currently be set with the DATATOOLKIT_STORE environment variable.

  Store path
  ----------

  The directory the the store is maintained in can be set via the store.path configuration
  parameter.

  config.store.path = "relative/to/datatoml"

  The system default is ~/.cache/julia/datatoolkit, which can be overriden with the
  DATATOOLKIT_STORE environment variable.

What do you make of that option?

kescobo commented 8 months ago

Oh yeah - that should work fine! Where is that documented? I went looking in a few places and didn't see it

tecosaur commented 8 months ago

You should be able to find it near the start of https://tecosaur.github.io/DataToolkitDocs/common/stable/plugins/store/, though in my comment above I pulled it from data> plugin info store :slightly_smiling_face:.

tecosaur commented 8 months ago

It also might just not have been there when you looked, it was added in 3783f1d15e32b2ba842917f22d5ce92b934becc5 a ~month ago.

tecosaur commented 8 months ago

:facepalm: I have just realised looking at that commit, the env var value will be baked in during precompilation. I'll fix that in 0.7.2 in the near future.

tecosaur commented 8 months ago

Should be properly implemented as of 1700c3e2ac34017a24caebd7754a75eaaa26ace4, and I'm tagging 0.7.2 now :slightly_smiling_face:

tecosaur commented 1 month ago

I just want to note here that I'm open to adding this as a Preference in the future, if there's demand for it.

kescobo commented 1 month ago

I demand you add it! But not really :-P