search

JuliaData / JuliaDB.jl

Parallel analytical database in pure Julia
http://juliadb.org/
Other
764 stars 61 forks source link

Pkg.test("JuliaDB") fails on Windows #166

Open bmharsha opened 6 years ago

bmharsha commented 6 years ago

Exact error message and versioninfo() can be found below

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.2 (2017-12-13 18:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> Pkg.test("JuliaDB")
INFO: Testing JuliaDB
Test Summary: | Pass  Total
iteration     |   20     20
Test Summary: | Pass  Total
Utilities     |    3      3
Test Summary: | Pass  Total
Interval      |   10     10
Test Summary: | Pass  Total
table         |  112    112
Test Summary: | Pass  Total
map & reduce  |    2      2
Test Summary: | Pass  Total
getindex      |   30     30
Test Summary: | Pass  Total
select        |    6      6
Test Summary: | Pass  Total
convertdim    |    9      9
Test Summary: | Pass  Total
reducedim     |   20     20
Test Summary: | Pass  Total
select        |    1      1
Test Summary: | Pass  Total
permutedims   |    5      5
Test Summary: | Pass  Total
mapslices     |   11     11
Test Summary: | Pass  Total
flatten       |    3      3
Test Summary: | Pass  Total
join          |  175    175
Test Summary: | Pass  Total
broadcast     |   32     32
Test Summary: | Pass  Total
setindex!     |    4      4
Test Summary: | Pass  Total
extractarray  |    9      9
Test Summary: | Pass  Total
printing      |    1      1
Test Summary: | Pass  Total
has_overlaps  |    6      6
Test Summary: | Pass  Total
with_overlaps |    5      5
Test Summary: | Pass  Total
subtable      |    7      7
Test Summary:       | Pass  Total
Iterators.partition |    1      1
Test Summary: | Pass  Total
rechunk       |    1      1
ERROR (unhandled task failure): KeyError: key 2553 not found
getindex at .\dict.jl:474 [inlined]
datastore_delete(::Int64) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:123
pooldelete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:147 [inlined]
free!(::MemPool.DRef) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\chunks.jl:110
(::Dagger.##92#93)() at .\task.jl:335
Stacktrace:
 [1] sync_end() at .\task.jl:287
 [2] macro expansion at .\task.jl:303 [inlined]
 [3] free_chunks(::Tuple{Dagger.Chunk{Array{Float64,1},MemPool.DRef},Dagger.ArrayDomain{1}}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [4] macro expansion at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:129 [inlined]
 [5] macro expansion at .\task.jl:302 [inlined]
 [6] free_chunks(::Array{Union{Dagger.Chunk, Dagger.Thunk},1}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [7] (::Dagger.##94#95{Dagger.DArray{Float64,1,Base.#cat}})() at .\event.jl:73
ERROR (unhandled task failure): KeyError: key 2551 not found
getindex at .\dict.jl:474 [inlined]
datastore_delete(::Int64) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:123
pooldelete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:147 [inlined]
free!(::MemPool.DRef) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\chunks.jl:110
(::Dagger.##92#93)() at .\task.jl:335
Stacktrace:
 [1] sync_end() at .\task.jl:287
 [2] macro expansion at .\task.jl:303 [inlined]
 [3] free_chunks(::Tuple{Dagger.Chunk{Array{Float64,1},MemPool.DRef},Dagger.ArrayDomain{1}}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [4] macro expansion at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:129 [inlined]
 [5] macro expansion at .\task.jl:302 [inlined]
 [6] free_chunks(::Array{Union{Dagger.Chunk, Dagger.Thunk},1}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [7] (::Dagger.##94#95{Dagger.DArray{Float64,1,Base.#cat}})() at .\event.jl:73
ERROR (unhandled task failure): On worker 3:
KeyError: key 2043 not found
getindex at .\dict.jl:474 [inlined]
datastore_delete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:123
pooldelete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:147
#106 at .\distributed\process_messages.jl:268 [inlined]
run_work_thunk at .\distributed\process_messages.jl:56
macro expansion at .\distributed\process_messages.jl:268 [inlined]
#105 at .\event.jl:73
#remotecall_fetch#141(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.Worker, ::MemPool.DRef, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:354
remotecall_fetch(::Function, ::Base.Distributed.Worker, ::MemPool.DRef, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:346
#remotecall_fetch#144(::Array{Any,1}, ::Function, ::Function, ::Int64, ::MemPool.DRef, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:367
pooldelete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:145 [inlined]
free!(::MemPool.DRef) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\chunks.jl:110
(::Dagger.##92#93)() at .\task.jl:335
Stacktrace:
 [1] sync_end() at .\task.jl:287
 [2] macro expansion at .\task.jl:303 [inlined]
 [3] free_chunks(::Tuple{Dagger.Chunk{Array{Float64,1},MemPool.DRef},Dagger.ArrayDomain{1}}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [4] macro expansion at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:129 [inlined]
 [5] macro expansion at .\task.jl:302 [inlined]
 [6] free_chunks(::Array{Union{Dagger.Chunk, Dagger.Thunk},1}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [7] (::Dagger.##94#95{Dagger.DArray{Float64,1,Base.#cat}})() at .\event.jl:73
ERROR (unhandled task failure): On worker 3:
KeyError: key 2040 not found
getindex at .\dict.jl:474 [inlined]
datastore_delete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:123
pooldelete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:147
#106 at .\distributed\process_messages.jl:268 [inlined]
run_work_thunk at .\distributed\process_messages.jl:56
macro expansion at .\distributed\process_messages.jl:268 [inlined]
#105 at .\event.jl:73
#remotecall_fetch#141(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.Worker, ::MemPool.DRef, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:354
remotecall_fetch(::Function, ::Base.Distributed.Worker, ::MemPool.DRef, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:346
#remotecall_fetch#144(::Array{Any,1}, ::Function, ::Function, ::Int64, ::MemPool.DRef, ::Vararg{Any,N} where N) at .\distributed\remotecall.jl:367
pooldelete at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\MemPool\src\datastore.jl:145 [inlined]
free!(::MemPool.DRef) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\chunks.jl:110
(::Dagger.##92#93)() at .\task.jl:335
Stacktrace:
 [1] sync_end() at .\task.jl:287
 [2] macro expansion at .\task.jl:303 [inlined]
 [3] free_chunks(::Tuple{Dagger.Chunk{Array{Float64,1},MemPool.DRef},Dagger.ArrayDomain{1}}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [4] macro expansion at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:129 [inlined]
 [5] macro expansion at .\task.jl:302 [inlined]
 [6] free_chunks(::Array{Union{Dagger.Chunk, Dagger.Thunk},1}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\Dagger\src\array\darray.jl:123
 [7] (::Dagger.##94#95{Dagger.DArray{Float64,1,Base.#cat}})() at .\event.jl:73
Test Summary: | Pass  Total
rechunk by    |    2      2
Test Summary:              | Pass  Total
PooledArray/DataValueArray |    2      2
Test Summary: | Pass  Total
Columns       |    2      2
Test Summary: | Pass  Total
ndsparse      |    1      1
Test Summary: | Pass  Total
table         |    2      2
ERROR: LoadError: LoadError: SystemError (with C:\Users\harsha\AppData\Local\Temp\jl_6BF3.tmp): mkdir: File exists
Stacktrace:
 [1] #systemerror#44 at .\error.jl:64 [inlined]
 [2] (::Base.#kw##systemerror)(::Array{Any,1}, ::Base.#systemerror, ::Symbol, ::Bool) at .\<missing>:0
 [3] mkdir(::String, ::UInt16) at .\file.jl:99
 [4] #fromchunks#49(::String, ::Int64, ::Array{Any,1}, ::Function, ::Array{Dagger.Thunk,1}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\JuliaDB\src\table.jl:152
 [5] (::JuliaDB.#kw##fromchunks)(::Array{Any,1}, ::JuliaDB.#fromchunks, ::Array{Dagger.Thunk,1}) at .\<missing>:0
 [6] #_loadtable#219(::Int64, ::String, ::Bool, ::UnitRange{Int64}, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::Type{T} where T, ::Array{String,1}) at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\JuliaDB\src\io.jl:142
 [7] (::JuliaDB.#kw##_loadtable)(::Array{Any,1}, ::JuliaDB.#_loadtable, ::Type{T} where T, ::Array{String,1}) at .\<missing>:0
 [8] #loadndsparse#218 at C:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\JuliaDB\src\io.jl:84 [inlined]
 [9] (::JuliaDB.#kw##loadndsparse)(::Array{Any,1}, ::JuliaDB.#loadndsparse, ::Array{String,1}) at .\<missing>:0
 [10] include_from_node1(::String) at .\loading.jl:576
 [11] include(::String) at .\sysimg.jl:14
 [12] include_from_node1(::String) at .\loading.jl:576
 [13] include(::String) at .\sysimg.jl:14
 [14] process_options(::Base.JLOptions) at .\client.jl:305
 [15] _start() at .\client.jl:371
while loading C:\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\JuliaDB\test\test_readwrite.jl, in expression starting on line 57
while loading C:\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\JuliaDB\test\runtests.jl, in expression starting on line 17
===============================[ ERROR: JuliaDB ]===============================

failed process: Process(`'C:\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\Julia-0.6.2\bin\julia.exe' -Cx86-64 '-JC:\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\Julia-0.6.2\lib\julia\sys.dll' --compile=yes --depwarn=yes --check-bounds=yes --code-coverage=none --color=yes --compilecache=yes 'C:\Users\harsha\AppData\Local\JuliaPro-0.6.2.2\pkgs-0.6.2.2\v0.6\JuliaDB\test\runtests.jl'`, ProcessExited(1)) [1]

================================================================================
ERROR: JuliaDB had test errors

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17* (2017-12-13 18:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

This issue occurred on JuliaDB v0.7.2

bmharsha commented 6 years ago

Just a FYI:

ERROR: LoadError: LoadError: SystemError (with C:\Users\harsha\AppData\Local\Temp\jl_6BF3.tmp): mkdir: File exists

I cleared the contents of C:\Users\harsha\AppData\Local\Temp and reran Pkg.test("JuliaDB") after restarting Julia, I encountered the same error.

I'm able to reproduce this error on JuliaDB master branch too, as of now, master branch is on following commit: ecf0f85bad640381e552d903dc16857cf3305efc

StefanKarpinski commented 6 years ago

That's an awful lot of backslashes. Related? Or just an incidental problem?

bmharsha commented 6 years ago

That's an awful lot of backslashes

I was using JuliaPro, hence the backslashes

Related? Or just an incidental problem?

It shouldn't be related, because, I'm able to reproduce this error on an independent Julia (From JuliaLang) installation too (Proof)

bmharsha commented 6 years ago 
 ERROR: LoadError: LoadError: SystemError (with C:\Users\harsha\AppData\Local\Temp\jl_6BF3.tmp): mkdir: File exists

We can fix above issue by replacing tempname() with joinpath(tempdir(), randstring()) in test_readwrite.jl, this issue is occuring because tempname() is creating a file in Windows temporary directory and returning it’s path, JuliaDB is trying to create a directory using the same path.

I made above changes locally and encountered following error: https://github.com/JuliaComputing/JuliaDB.jl/issues/168 as the test progressed, because we are using rm(n, recursive=true) in following line: https://github.com/JuliaComputing/JuliaDB.jl/blob/master/test/test_readwrite.jl#L122