"Access is denied." error with SharedArrays on Windows

[jtackm]

My package integration tests crash on Windows when using SharedArrays:

Testing learning.jl
(this can take a couple of minutes)
Test Summary: | Pass  Total
univar nbrs   |    5      5
┌ Warning: Skipping normalization, only experts should choose this option
└ @ FlashWeave D:\a\FlashWeave.jl\FlashWeave.jl\src\learning.jl:493
┌ Warning: Skipping normalization, only experts should choose this option
└ @ FlashWeave D:\a\FlashWeave.jl\FlashWeave.jl\src\learning.jl:493
┌ Warning: Skipping normalization, only experts should choose this option
└ @ FlashWeave D:\a\FlashWeave.jl\FlashWeave.jl\src\learning.jl:493
┌ Warning: Skipping normalization, only experts should choose this option
└ @ FlashWeave D:\a\FlashWeave.jl\FlashWeave.jl\src\learning.jl:493
Test Summary:     | Pass  Total
meta conditioning |    4      4
parallel multi_il: Error During Test at D:\a\FlashWeave.jl\FlashWeave.jl\test\learning.jl:187
  Got exception outside of a @test
  SystemError: mmap: Access is denied. 
  Stacktrace:
    [1] windowserror(p::Symbol, code::UInt32; extrainfo::Nothing)
      @ Base .\error.jl:183
    [2] #windowserror#64
      @ .\error.jl:182 [inlined]
    [3] windowserror
      @ .\error.jl:182 [inlined]
    [4] mmap(io::Mmap.Anonymous, ::Type{Vector{Float64}}, dims::Tuple{Int64}, offset::Int64; grow::Bool, shared::Bool)
      @ Mmap C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Mmap\src\Mmap.jl:224
    [5] mmap(io::Mmap.Anonymous, ::Type{Vector{Float64}}, dims::Tuple{Int64}, offset::Int64)
      @ Mmap C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Mmap\src\Mmap.jl:189
    [6] _shm_mmap_array(T::Type, dims::Tuple{Int64}, shm_seg_name::String, mode::UInt16)
      @ SharedArrays C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\SharedArrays\src\SharedArrays.jl:672
    [7] shm_mmap_array(T::Type, dims::Tuple{Int64}, shm_seg_name::String, mode::UInt16)
      @ SharedArrays C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\SharedArrays\src\SharedArrays.jl:651
    [8] SharedArrays.SharedVector{Float64}(dims::Tuple{Int64}; init::Bool, pids::Vector{Int64})
      @ SharedArrays C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\SharedArrays\src\SharedArrays.jl:118
    [9] SharedArray
      @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\SharedArrays\src\SharedArrays.jl:105 [inlined]
   [10] SharedArray
      @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\SharedArrays\src\SharedArrays.jl:369 [inlined]
   [11] pw_univar_neighbors(data::SparseMatrixCSC{Int64, Int64}; test_name::String, alpha::Float64, hps::Int64, n_obs_min::Int64, FDR::Bool, levels::Vector{Int64}, max_vals::Vector{Int64}, parallel::String, workers_local::Bool, cor_mat::Matrix{Float32}, tmp_folder::String, correct_reliable_only::Bool, shuffle_jobs::Bool, pmap_batch_size::Nothing)

which corresponds to the following code line within the package:

shared_pvals = SharedArray{Float64}(pvals)

Which tries to convert a Vector{Float64} to a SharedArray{Float64}. Parameters used for github CI: windows-latest, Julia version: 1 (which should expand to the latest stable release). For version: 1.2 everything runs fine.

Sorry for not providing a detailed MWE, I don't have a windows machine myself available right now.

[inkydragon]

Does this problem always occur on CI? Or maybe this problem has a certain probability of occurring.

I am able to replicate the problem about 30% of the time.

—— Mmap access denied with SharedArrays - General Usage / Performance - JuliaLang

I can't reproduce this issue locally at the moment. Maybe the probability of this problem appearing on my computer is less than 1% ~30%~.

julia> versioninfo()
Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

I'am trying to call make_network many times (10h+). But nothing happend. ensure julia 1.2 compat · meringlab/FlashWeave.jl@815bc8a

test code

Add those codes after [FlashWeave.jl/learning.jl](https://github.com/meringlab/FlashWeave.jl/blob/master/test/learning.jl#L111). ```julia function test_sa() for test_name in ["mi", "mi_nz", "fz", "fz_nz"], max_k in [0, 3], make_sparse in [true, false], parallel in ["single", "single_il", "multi_il"] # if max_k==0 && parallel == "single_il" continue end graph_name = "exp_$(test_name)_maxk$(max_k)" exp_graph = exp_dict[graph_name] testset_name = "$graph_name: sparse=$make_sparse, parallel=$parallel" @testset "testset_name" begin is_il = endswith(parallel, "_il") time_limit = is_il ? 30.0 : 0.0 pred_graph = @silence_stdout begin make_network(data, test_name, make_sparse, 64, max_k=max_k, parallel=parallel, time_limit=time_limit) end end end end while true try test_sa() catch e @show e println("$(now()): catched error") break end end ```

[jtackm]

Thanks a lot for looking into this. I re-ran CI 3 times now, always the error message. I have a windows machine at home, will try that as well tonight.

[ViralBShah]

Looks like this is with Julia 1.6

[jtackm]

Yes, this is with julia 1.6.3. I'm consistently (5 out of 5 tries) reproducing the error when manually running package tests on my home Windows 10 system (should have latest updates, but I can provide specifics if necessary) with that julia version.

[ViralBShah]

Do you have some special OS protections enabled?

[jtackm]

I didn't tinker with the system, things should be at defaults (in line with windows-latest on github CI, which shows the same error). But let me know if I should check something in particular.

[ViralBShah]

Does it happen on your windows machine at home too?

[jtackm]

Yes, same on both systems. On my home machine, I can generate SharedArrays after adding a single worker via Distributed (similar to what is done in the package tests) in a REPL session without error, but the package tests (]test FlashWeave) fail.

Code that works:

using Distributed, SharedArrays
addprocs(1)
xs = rand(100)
xs_shared = SharedArray{Float64}(xs)

versioninfo of my home machine:

Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

[ViralBShah]

And what if you try one of the 1.7-rc releases?

[jtackm]

Tried 1.7.0-rc2, it works

[atbug]

I am seeing the same thing in my package:

https://github.com/Hop-developers/Hop.jl/runs/6604925977

[jtackm]

Can confirm that the error started reappearing ca. a week ago (julia 1.7.2): https://github.com/meringlab/FlashWeave.jl/runs/6523169674?check_suite_focus=true

Latest CI runs, all fails (julia 1.6.6 and 1.7.2) with "Access is denied" errors: https://github.com/meringlab/FlashWeave.jl/actions/runs/2384313185

[atbug]

Julia 1.8 seems to fix things for me: https://github.com/HopTB/HopTB.jl/runs/7927457499

[AsafManela]

Unfortunately, this still happens on windows in Julia 1.9.2.

See here for CI runs: https://github.com/AsafManela/HurdleDMR.jl/actions/runs/5875384448/job/15931582566#step:6:388

I also tried this on a Windows 10 local machine I have and got the same error.

[montyvesselinov]

I confirm this error with julia 1.10.0

ERROR: LoadError: SystemError: mmap: Access is denied. 
Stacktrace:
  [1] windowserror(p::Symbol, code::UInt32; extrainfo::Nothing)
    @ Base .\error.jl:191
  [2] kwcall(::NamedTuple, ::typeof(Base.windowserror), p::Any)
    @ Base .\error.jl:190 [inlined]
  [3] mmap(io::Mmap.Anonymous, ::Type{Matrix{Float64}}, dims::Tuple{Int64, Int64}, offset::Int64; grow::Bool, shared::Bool)
    @ Mmap C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Mmap\src\Mmap.jl:230
  [4] mmap(io::Mmap.Anonymous, ::Type{Matrix{Float64}}, dims::Tuple{Int64, Int64}, offset::Int64)
    @ Mmap C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Mmap\src\Mmap.jl:186
  [5] _shm_mmap_array(T::Type, dims::Tuple{Int64, Int64}, shm_seg_name::String, mode::UInt16)
    @ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:672
  [6] shm_mmap_array(T::Type, dims::Tuple{Int64, Int64}, shm_seg_name::String, mode::UInt16)
    @ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:651
  [7] SharedArrays.SharedMatrix{Float64}(dims::Tuple{Int64, Int64}; init::Bool, pids::Vector{Int64})
    @ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:118
  [8] SharedArray
    @ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:104 [inlined]
  [9] SharedArray
    @ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:161 [inlined]
 [10] (SharedArrays.SharedArray{Float64})(m::Int64, n::Int64)
    @ SharedArrays C:\Users\monty\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\share\julia\stdlib\v1.10\SharedArrays\src\SharedArrays.jl:169
 [11] forward(madsdata::Dict{…}, paramarray::Matrix{…}; parallel::Bool, all::Bool, checkpointfrequency::Int64, checkpointfilename::String)
    @ Mads C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:106
 [12] forward
    @ C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:48 [inlined]
 [13] emceesampling(madsdata::Dict{…}, p0::Matrix{…}; filename::String, load::Bool, save::Bool, execute::Bool, numwalkers::Int64, nsteps::Int64, burnin::Int64, thinning::Int64, seed::Int64, weightfactor::Float64, rng::Nothing, distributed_function::Bool)
    @ Mads C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:147
 [14] emceesampling
    @ Mads C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:106
 [12] forward
    @ C:\Users\monty\.julia\dev\Mads\src\MadsForward.jl:48 [inlined]
    @ Mads C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:147
 [14] emceesampling
    @ C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:116 [inlined]
 [15] emceesampling(madsdata::Dict{…}; filename::String, load::Bool, save::Bool, execute::Bool, numwalkers::Int64, sigma::Float64, seed::Int64, rng::Nothing, kw::@Kwargs{…})
    @ Mads C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:112
 [16] emceesampling
    @ C:\Users\monty\.julia\dev\Mads\src\MadsMonteCarlo.jl:80 [inlined]
 [17] emcee(md::Dict{…}, start_date::Dates.Date; parammin::Matrix{…}, parammax::Matrix{…}, madsdir::String, case::String, casename::String, extracasename::String, f_emcee_pi::String, f_emcee_of::String, f_emcee_pi_top::String, f_emcee_parameters::String, f_emcee_parameters_mean::String, f_emcee_parameters_jld::String, f_emcee_scatter::String, f_emcee_scatter_all::String, f_emcee_spaghetti::String, f_emcee_spaghetti_all::String, f_emcee_best_worst::String, f_emcee_p10_50_90::String, f_emcee_p10_50_90_csv::String, f_emcee_all_p10_50_90::String, f_emcee_all_p10_50_90_csv::String, f_emcee_min_max::String, f_emcee_min_max_csv::String, f_emcee_all_min_max::String, f_emcee_all_min_max_csv::String, ofmax_ratio::Float64, ncases::Int64, nsteps::Int64, burnin::Int64, thinning::Int64, numwalkers::Int64, load::Bool, save::Bool, plot::Bool, plot_scatter::Bool, plot_all_scatter::Bool, plot_spaghetti::Bool, plot_all_spaghetti::Bool, execute::Bool, best_worst::Int64, top::Int64, md_prediction::Dict{…}, predictions::Matrix{…}, spaghetti_grayscale::Bool, spaghetti_opacity::Float64)
    @ GIMI C:\Users\monty\.julia\dev\GIMI\src\GIMI_mads.jl:630

[montyvesselinov]

it occurs when for a second time a function containing:

r = SharedArrays.SharedArray{Float64}(length(rv1), ncases)

is called.