search

oxinabox / DataDeps.jl

reproducible data setup for reproducible science
Other
151 stars 43 forks source link

Deleting the folder `.julia/datadeps/` breaks the package until the next time its precompiled #29

Open Evizero opened 6 years ago

Evizero commented 6 years ago

This is because this code https://github.com/oxinabox/DataDeps.jl/blob/master/src/locations.jl#L19 (i.e. mkpath(first(default_loadpath))) is only executed during precompilation.

So if someone deletes the whole .julia/datadeps subfolder for some reason (which I just did because I wanted to retrigger all downloads for testing), it breaks the package by throwing the following error

julia> datadep"MNIST"
ERROR: No possible save path
Stacktrace:
 [1] determine_save_path(::String, ::Void) at /home/csto/.julia/v0.6/DataDeps/src/locations.jl:144
 [2] handle_missing(::DataDeps.DataDep{String,Array{String,1},Base.#download,Base.#identity}, ::Void) at /home/csto/.julia/v0.6/DataDeps/src/resolution_automatic.jl:2
 [3] _resolve(::DataDeps.DataDep{String,Array{String,1},Base.#download,Base.#identity}, ::Void) at /home/csto/.julia/v0.6/DataDeps/src/DataDeps.jl:78
 [4] resolve(::DataDeps.DataDep{String,Array{String,1},Base.#download,Base.#identity}, ::String, ::Void) at /home/csto/.julia/v0.6/DataDeps/src/DataDeps.jl:51

a possible solution would be to move that code into __init__

oxinabox commented 6 years ago

The other option (which I have been tossing up) is going the other way. Explictly only creating ~/.julia/datadeps during Pkg.build("DataDeps"), (whih is effectively where it is now, inplictly) and if no folder in the DATADEPS_LOADPATH (which defaults to a long list starting with ~/.julia/datadeps) exists, give a nice Error saying at least one folder on the load path must exist and be writable.

Maybe continue with DataDeps.jl having the ability to recover from errors by having a dialog "Please create one of these folders and Retry. [Retry/Abort]?" Or even a menu listing them all and asking which we should create / retry/ abort

I feel like deleting things that are on the default loadpath is a clear signal the user may not want to save there.

Evizero commented 6 years ago

fair points

oxinabox commented 6 years ago

I think, perhaps what should be done is is there are no writeable directories in the loadpath, a notice should be displayed saying. "Please ensure at least one of the directories on the loadpath exists and is writeable: ... [List them ]. Or set the loadpath environment variable"