Open jtackm opened 2 years ago
Does this problem always occur on CI? Or maybe this problem has a certain probability of occurring.
I am able to replicate the problem about 30% of the time.
—— Mmap access denied with SharedArrays - General Usage / Performance - JuliaLang
I can't reproduce this issue locally at the moment. Maybe the probability of this problem appearing on my computer is less than 1% ~30%~.
julia> versioninfo()
Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
I'am trying to call make_network
many times (10h+). But nothing happend.
ensure julia 1.2 compat · meringlab/FlashWeave.jl@815bc8a
Thanks a lot for looking into this. I re-ran CI 3 times now, always the error message. I have a windows machine at home, will try that as well tonight.
Looks like this is with Julia 1.6
Yes, this is with julia 1.6.3. I'm consistently (5 out of 5 tries) reproducing the error when manually running package tests on my home Windows 10 system (should have latest updates, but I can provide specifics if necessary) with that julia version.
Do you have some special OS protections enabled?
I didn't tinker with the system, things should be at defaults (in line with windows-latest
on github CI, which shows the same error). But let me know if I should check something in particular.
Does it happen on your windows machine at home too?
Yes, same on both systems. On my home machine, I can generate SharedArrays after adding a single worker via Distributed (similar to what is done in the package tests) in a REPL session without error, but the package tests (]test FlashWeave
) fail.
Code that works:
using Distributed, SharedArrays
addprocs(1)
xs = rand(100)
xs_shared = SharedArray{Float64}(xs)
versioninfo
of my home machine:
Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
And what if you try one of the 1.7-rc releases?
Tried 1.7.0-rc2, it works
I am seeing the same thing in my package:
Can confirm that the error started reappearing ca. a week ago (julia 1.7.2): https://github.com/meringlab/FlashWeave.jl/runs/6523169674?check_suite_focus=true
Latest CI runs, all fails (julia 1.6.6 and 1.7.2) with "Access is denied" errors: https://github.com/meringlab/FlashWeave.jl/actions/runs/2384313185
Julia 1.8 seems to fix things for me: https://github.com/HopTB/HopTB.jl/runs/7927457499
Unfortunately, this still happens on windows in Julia 1.9.2.
See here for CI runs: https://github.com/AsafManela/HurdleDMR.jl/actions/runs/5875384448/job/15931582566#step:6:388
I also tried this on a Windows 10 local machine I have and got the same error.
I confirm this error with julia 1.10.0
ERROR: LoadError: SystemError: mmap: Access is denied.
Stacktrace:
[1] windowserror(p::Symbol, code::UInt32; extrainfo::Nothing)
@ Base .\error.jl:191
[2] kwcall(::NamedTuple, ::typeof(Base.windowserror), p::Any)
@ Base .\error.jl:190 [inlined]
[3] mmap(io::Mmap.Anonymous, ::Type{Matrix{Float64}}, dims::Tuple{Int64, Int64}, offset::Int64; grow::Bool, shared::Bool)
@ Mmap C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Mmap\src\Mmap.jl:230
[4] mmap(io::Mmap.Anonymous, ::Type{Matrix{Float64}}, dims::Tuple{Int64, Int64}, offset::Int64)
@ Mmap C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Mmap\src\Mmap.jl:186
[5] _shm_mmap_array(T::Type, dims::Tuple{Int64, Int64}, shm_seg_name::String, mode::UInt16)
@ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:672
[6] shm_mmap_array(T::Type, dims::Tuple{Int64, Int64}, shm_seg_name::String, mode::UInt16)
@ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:651
[7] SharedArrays.SharedMatrix{Float64}(dims::Tuple{Int64, Int64}; init::Bool, pids::Vector{Int64})
@ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:118
[8] SharedArray
@ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:104 [inlined]
[9] SharedArray
@ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:161 [inlined]
[10] (SharedArrays.SharedArray{Float64})(m::Int64, n::Int64)
@ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:169
[11] forward(madsdata::Dict{…}, paramarray::Matrix{…}; parallel::Bool, all::Bool, checkpointfrequency::Int64, checkpointfilename::String)
@ Mads C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:106
[12] forward
@ C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:48 [inlined]
[13] emceesampling(madsdata::Dict{…}, p0::Matrix{…}; filename::String, load::Bool, save::Bool, execute::Bool, numwalkers::Int64, nsteps::Int64, burnin::Int64, thinning::Int64, seed::Int64, weightfactor::Float64, rng::Nothing, distributed_function::Bool)
@ Mads C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:147
[14] emceesampling
@ Mads C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:106
[12] forward
@ C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:48 [inlined]
@ Mads C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:147
[14] emceesampling
@ C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:116 [inlined]
[15] emceesampling(madsdata::Dict{…}; filename::String, load::Bool, save::Bool, execute::Bool, numwalkers::Int64, sigma::Float64, seed::Int64, rng::Nothing, kw::@Kwargs{…})
@ Mads C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:112
[16] emceesampling
@ C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:80 [inlined]
[17] emcee(md::Dict{…}, start_date::Dates.Date; parammin::Matrix{…}, parammax::Matrix{…}, madsdir::String, case::String, casename::String, extracasename::String, f_emcee_pi::String, f_emcee_of::String, f_emcee_pi_top::String, f_emcee_parameters::String, f_emcee_parameters_mean::String, f_emcee_parameters_jld::String, f_emcee_scatter::String, f_emcee_scatter_all::String, f_emcee_spaghetti::String, f_emcee_spaghetti_all::String, f_emcee_best_worst::String, f_emcee_p10_50_90::String, f_emcee_p10_50_90_csv::String, f_emcee_all_p10_50_90::String, f_emcee_all_p10_50_90_csv::String, f_emcee_min_max::String, f_emcee_min_max_csv::String, f_emcee_all_min_max::String, f_emcee_all_min_max_csv::String, ofmax_ratio::Float64, ncases::Int64, nsteps::Int64, burnin::Int64, thinning::Int64, numwalkers::Int64, load::Bool, save::Bool, plot::Bool, plot_scatter::Bool, plot_all_scatter::Bool, plot_spaghetti::Bool, plot_all_spaghetti::Bool, execute::Bool, best_worst::Int64, top::Int64, md_prediction::Dict{…}, predictions::Matrix{…}, spaghetti_grayscale::Bool, spaghetti_opacity::Float64)
@ GIMI C:\Users\monty\.julia\dev\GIMI\src\GIMI_mads.jl:630
it occurs when for a second time a function containing:
r = SharedArrays.SharedArray{Float64}(length(rv1), ncases)
is called.
My package integration tests crash on Windows when using SharedArrays:
which corresponds to the following code line within the package:
Which tries to convert a Vector{Float64} to a SharedArray{Float64}. Parameters used for github CI:
windows-latest
, Juliaversion: 1
(which should expand to the latest stable release). Forversion: 1.2
everything runs fine.Sorry for not providing a detailed MWE, I don't have a windows machine myself available right now.