JuliaParallel / Dagger.jl

A framework for out-of-core and parallel execution
Other
629 stars 67 forks source link

Update path splitting rule in Dagger.File / Dagger.tofile #425

Closed svilupp closed 1 year ago

svilupp commented 1 year ago

This PR proposes a fix to a potential bug discovered in https://github.com/JuliaParallel/DTables.jl/issues/35

Behaviour: When using relative paths (eg, Dagger.tofile(data, "data_processed/obj_$i.arrow", ...), the directories would be ignored and files would be saved directly in pwd().

Reason: Path splitting was done with isabspath (see link, but that doesn't work for relative paths.

Example:

path = "data_output/out.arrow"
isabspath(path) # false! not helpful if we want to know if we need to use pwd() to set directory!

Proposed change:

dir, file = dirname(path), basename(path)
dir = isempty(dir) ? pwd() : dir

I've also added a test case for the relative paths to prevent future regressions. Current testing used tempdir(), which provides an absolute path; hence, the bug wasn't observed.

jpsamaroo commented 1 year ago

Thanks for finding and fixing this!