JuliaDB.save and JuliaDB.load operations are locking the file on Windows

[bmharsha]

Here are the minimal steps that can be used to reproduce this issue on Windows

• Execute following code and copy the path of variable n

  using JuliaDB
  t = table([1,2,3,4], [1,2,3,4], chunks=2)
  n = joinpath(tempdir(), randstring())
  x = JuliaDB.save(t, n)
  @show n
  rm(n, recursive=true, force=true)

  After executing above code, you will encounter following error, this is to illustrate locking in JuliaDB.save

  ERROR: unlink: operation not permitted (EPERM)
  Stacktrace:
  [1] uv_error at .\libuv.jl:68 [inlined]
  [2] unlink(::String) at .\file.jl:500
  [3] #rm#7(::Bool, ::Bool, ::Function, ::String) at .\file.jl:145
  [4] (::Base.Filesystem.#kw##rm)(::Array{Any,1}, ::Base.Filesystem.#rm, ::String) at .\<missing>:0
  [5] #rm#7(::Bool, ::Bool, ::Function, ::String) at .\file.jl:155
  [6] (::Base.Filesystem.#kw##rm)(::Array{Any,1}, ::Base.Filesystem.#rm, ::String) at .\<missing>:0

• You can restart Julia and load the file that we created in above code and then try to delete this file

  #n="C:\\Users\\myusername\\AppData\\Local\\Temp\\FJwetrhV"
  using JuliaDB
  t1 = JuliaDB.load(n)   
  rm(n, recursive=true, force=true)

  After executing above code, you will once again encounter the same unlink: operation not permitted (EPERM) error. If you try to delete the contents of your temporary directory (While Julia REPL / process is still running) using Windows GUI, you will encounter following error

You will be able to delete above files if you quit your Julia process.

[MaximilianJHuber]

JuliaDB memory-maps the table to the file. Why would you want to delete it?

[bmharsha]

Why would you want to delete it?

I was trying to fix https://github.com/JuliaComputing/JuliaDB.jl/issues/166 and encountered this error as the test progressed (More details in this comment), I felt this might be a bug, so reported this issue.

[deo1]

JuliaDB memory-maps the table to the file. Why would you want to delete it?

A scenario I can think of is when you are saving data to disk for long term storage, but may overwrite the version on disk within the same Julia process lifetime. A common pattern is e.g.

files:

• 2019_12_25.bin
• 2019_12_26.bin
• etc...

You may load, merge, overwrite multiple times during the process lifetime. This is the ETL pipeline scenario, not the (potentially) out-of-memory analytics scenario.

A simple workaround that I have used is to remove the load(f) from src/io.jl save(data::Dataset, f::AbstractString) with a more specific dispatch signature.

e.g.

import JuliaDB.save
using MemPool

function JuliaDB.save(data::IndexedTable, f::AbstractString)
    sz = open(f, "w") do io
        MemPool.serialize(io, MemPool.MMWrap(data))
    end
    # load(f)  # remove this
end

Apologies, after messing around a bit more, this doesn't cover the scenario where you save, load, and save again. The simplest workaround for this seems to be to just use the low level serialize and deserialize methods directly.

e.g.

using JuliaDB, MemPool

f = "./data/test/scratch.bin"
t1 = table((a = [1,2,3,4], b = [1,3,6,6]), pkey=:b)

MemPool.serialize(f, t1)
t2 = MemPool.deserialize(f)
MemPool.serialize(f, merge(t1, t2))
t3 = MemPool.deserialize(f)

@assert t3 == merge(t1, t2)