search

gher-uliege / DIVAnd.jl

DIVAnd performs an n-dimensional variational analysis of arbitrarily located observations
GNU General Public License v2.0
70 stars 11 forks source link

Unhandled Task ERROR: EOFError: read end of file #127

Closed ctroupin closed 1 year ago

ctroupin commented 1 year ago

I'm running a fairly large DIVAndapplication,

sz = (461, 237, 102)

with 10⁶ - 10⁷ observations. The target horizontal resolution is 0.25°, but it lead to the error copied below. Meanwhile, decreasing the resolution to 0.5° allows the run to complete.

The code is run with the command:

julia -p 6 run_clim.jl

Error log

[ Info: Time step 1 / 12
[ Info: analysis time index 1 uses the backgrond time index 1
[ Info: scaled correlation length (min,max) in dimension 1: (8328.553991071729, 332894.02035622543)
[ Info: scaled correlation length (min,max) in dimension 2: (15540.000000000042, 332999.40400347713)
[ Info: scaled correlation length (min,max) in dimension 3: (15.0, 500.0)
┌ Warning: resolution (23952.21208057843) is too coarse for correlation length 34335.30666772388 in dimension 1 at indices CartesianIndex(256, 27, 1) (skipping further tests). It is recommended that the resolution is at least 2 times finer than the correlation length.
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/utils.jl:18
┌ Warning: resolution (27798.770931080733) is too coarse for correlation length 53390.22087611618 in dimension 2 at indices CartesianIndex(259, 27, 1) (skipping further tests). It is recommended that the resolution is at least 2 times finer than the correlation length.
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/utils.jl:18
┌ Warning: resolution (15.0) is too coarse for correlation length 25.0 in dimension 3 at indices CartesianIndex(1, 1, 21) (skipping further tests). It is recommended that the resolution is at least 2 times finer than the correlation length.
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/utils.jl:18
┌ Debug: ("cutter", [(0.49262956170026395, 64.60692493785176), (0.5590175205417217, 11.978925429095922), (0.6, 5.0)], (461, 237, 102), [0.0, 0.0, 0.0], 100, :direct)
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_cutter.jl:39
┌ Debug: biggestproblemdirectn [6.25e6 4.0e6 2.0e6 3.90625e6]
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_fittocpu.jl:59
┌ Debug: problemcut: 11144214
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_fittocpu.jl:69
┌ Debug: biggestproblemdirect: 2.0e6
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_fittocpu.jl:70
┌ Debug: winsize: [461, 237, 6]
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_fittocpu.jl:170
┌ Debug: csteps [0, 0, 0]
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:83
┌ Debug: error method: cpme
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:109
[ Info: number of windows: 49
┌ Debug: mean_Labs: [122431.31365519034, 138295.6927727213, 176.67216975167463]
└ @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:113
Worker 4 terminated.
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:945
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:953
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484
Worker 3 terminated.
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:947
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:953
  [4] unsafe_read
    @ ./io.jl:759 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:758
  [6] read!
    @ ./io.jl:760 [inlined]
  [7] deserialize_hdr_raw
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:484
Worker 2 terminated.
Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:947
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:953
  [4] unsafe_read
    @ ./io.jl:759 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:758
  [6] read!
    @ ./io.jl:760 [inlined]
  [7] deserialize_hdr_raw
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:484
Unhandled Task ERROR: ProcessExitedException(2)

...and 2 more exceptions.

Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:436
 [2] (::Distributed.var"#177#179"{DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}})()
   @ Distributed ./task.jl:455
ERROR: LoadError: TaskFailedException

    nested task error: ProcessExitedException(2)

    ...and 2 more exceptions.

    Stacktrace:
     [1] sync_end(c::Channel{Any})
       @ Base ./task.jl:436
     [2] (::Distributed.var"#177#179"{DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}})()
       @ Distributed ./task.jl:455
Stacktrace:
  [1] sync_end(c::Channel{Any})
    @ Base ./task.jl:436
  [2] macro expansion
    @ ./task.jl:455 [inlined]
  [3] DIVAndgo(mask::BitArray{3}, pmn::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, xi::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, f::Vector{Float64}, Labs::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}, errormethod::Symbol; moddim::Vector{Float64}, velocity::Tuple{}, MEMTOFIT::Int64, QCMETHOD::Tuple{}, RTIMESONESCALES::Tuple{}, solver::Symbol, overlapfactor::Float64, filteranom::Int64, filtererr::Int64, otherargs::Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:115
  [4] (::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), DIVAnd.var"#639#642"{DIVAnd.var"#639#640#643"{TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Bool, NCDatasets.CFVariable{Union{Missing, Float32}, 4, NCDatasets.Variable{Float32, 4, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :missing_values, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Tuple{Float32}, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:solver, :MEMTOFIT, :coeff_derivative2), Tuple{Symbol, Int64, Vector{Float64}}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Vector{Float64}, Int64})(ds::NCDataset{Nothing})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/diva.jl:644
  [5] NCDataset(::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), DIVAnd.var"#639#642"{DIVAnd.var"#639#640#643"{TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Bool, NCDatasets.CFVariable{Union{Missing, Float32}, 4, NCDatasets.Variable{Float32, 4, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :missing_values, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Tuple{Float32}, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:solver, :MEMTOFIT, :coeff_derivative2), Tuple{Symbol, Int64, Vector{Float64}}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Vector{Float64}, Int64}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/dataset.jl:241
  [6] NCDataset(::Function, ::String, ::Vararg{String})
    @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/dataset.jl:238
  [7] diva3d(xi::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Vector{Float64}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, value::Vector{Float64}, len::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}, filename::String, varname::String; datadir::String, bathname::String, bathisglobal::Bool, plotres::DIVAnd.var"#453#469", timeorigin::DateTime, moddim::Vector{Float64}, zlevel::Symbol, ncvarattrib::OrderedDict{String, String}, ncglobalattrib::OrderedDict{String, String}, transform::Tuple{DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}}, distfun::typeof(DIVAnd.distfun_m), mask::BitArray{3}, background::DIVAnd.var"#639#642"{DIVAnd.var"#639#640#643"{TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Bool, NCDatasets.CFVariable{Union{Missing, Float32}, 4, NCDatasets.Variable{Float32, 4, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :missing_values, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Tuple{Float32}, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, background_epsilon2_factor::Nothing, background_lenz::Nothing, background_len::Nothing, background_lenz_factor::Int64, filterbackground::Int64, fitcorrlen::Bool, fithorzcorrlen::Bool, fitvertcorrlen::Bool, fithorz_param::Dict{Any, Any}, fitvert_param::Dict{Any, Any}, memtofit::Int64, overlapfactor::Float64, niter_e::Int64, minfield::Float64, maxfield::Float64, surfextend::Bool, velocity::Tuple{}, stat_per_timeslice::Bool, error_thresholds::Vector{Tuple{String, Float64}}, divamethod::typeof(DIVAndgo), kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:solver, :MEMTOFIT, :coeff_derivative2), Tuple{Symbol, Int64, Vector{Float64}}}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/diva.jl:375
  [8] macro expansion
    @ ./timing.jl:262 [inlined]
  [9] top-level scope
    @ ~/EMODnet-Chemistry/julia/european-climatologies/background.jl:395
 [10] include
    @ ./client.jl:476 [inlined]
 [11] top-level scope
    @ ~/EMODnet-Chemistry/julia/european-climatologies/run_clim.jl:31
in expression starting at /home/ctroupin/EMODnet-Chemistry/julia/european-climatologies/background.jl:277
in expression starting at /home/ctroupin/EMODnet-Chemistry/julia/european-climatologies/run_clim.jl:21
ctroupin commented 1 year ago

Now if I run the same code with

julia run_clim.jl

it works in some cases, while in other cases (more observations I guess), it in the time step 4/12, with the following stacktrace:

Unhandled Task ERROR: SuiteSparse.CHOLMOD.CHOLMODException("out of memory")
Stacktrace:
  [1] error_handler(status::Int32, file::Cstring, line::Int32, message::Cstring)
    @ SuiteSparse.CHOLMOD ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:127
  [2] cholmod_l_factorize_p(arg1::SuiteSparse.CHOLMOD.Sparse{Float64}, arg2::Base.RefValue{Float64}, arg3::Ptr{Nothing}, arg4::Int64, arg5::SuiteSparse.CHOLMOD.Factor{Float64}, arg6::Base.RefValue{SuiteSparse.LibSuiteSparse.cholmod_common_struct})
    @ SuiteSparse.LibSuiteSparse ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/lib/x86_64-linux-gnu.jl:1116
  [3] factorize_p!(A::SuiteSparse.CHOLMOD.Sparse{Float64}, β::Float64, F::SuiteSparse.CHOLMOD.Factor{Float64})
    @ SuiteSparse.CHOLMOD ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:616
  [4] cholesky!(F::SuiteSparse.CHOLMOD.Factor{Float64}, A::SuiteSparse.CHOLMOD.Sparse{Float64}; shift::Float64, check::Bool)
    @ SuiteSparse.CHOLMOD ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1147
  [5] #cholesky#8
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1185 [inlined]
  [6] cholesky
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1178 [inlined]
  [7] #cholesky#9
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1297 [inlined]
  [8] cholesky
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1297 [inlined]
  [9] factorize!(C::CovarIS{Float64, SparseArrays.SparseMatrixCSC{Float64, Int64}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/special_matrices.jl:117
 [10] DIVAnd_factorize!(s::DIVAnd.DIVAnd_struct{Float64, Int64, 3, SparseArrays.SparseMatrixCSC{Float64, Int64}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_factorize.jl:40
 [11] DIVAndrun(operatortype::Type, mask::BitArray{3}, pmnin::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, xiin::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, f::Vector{Float64}, lin::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}; velocity::Tuple{}, primal::Bool, factorize::Bool, tol::Float64, maxit::Int64, minit::Int64, constraints::Tuple{}, inversion::Symbol, moddim::Vector{Float64}, fracindex::Matrix{Float64}, alpha::Vector{Any}, keepLanczosVectors::Int64, compPC::typeof(DIVAnd_pc_none), progress::DIVAnd.var"#286#288", fi0::Array{Float64, 3}, f0::Vector{Float64}, alphabc::Float64, scale_len::Bool, btrunc::Vector{Any}, MEMTOFIT::Int64, topographyforfluxes::Tuple{}, fluxes::Tuple{}, epsfluxes::Int64, epsilon2forfractions::Int64, RTIMESONESCALES::Tuple{}, QCMETHOD::Tuple{}, coeff_laplacian::Vector{Float64}, coeff_derivative2::Vector{Float64}, mean_Labs::Vector{Float64})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndrun.jl:125
 [12] #DIVAndrun#290
    @ ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndrun.jl:298 [inlined]
 [13] DIVAnd_cpme(mask::BitArray{3}, pmn::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, xi::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, f::Vector{Float64}, Labs::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}; csteps::Vector{Int64}, lmask::Vector{Any}, alphapc::Vector{Any}, mean_Labs::Vector{Float64}, otherargs::Base.Pairs{Symbol, Any, NTuple{6, Symbol}, NamedTuple{(:moddim, :MEMTOFIT, :QCMETHOD, :RTIMESONESCALES, :velocity, :coeff_derivative2), Tuple{Vector{Float64}, Int64, Tuple{}, Tuple{}, Tuple{}, Vector{Float64}}}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_cpme.jl:88
 [14] macro expansion
    @ ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:281 [inlined]
 [15] (::DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool})(R::UnitRange{Int64}, lo::Int64, hi::Int64)
    @ DIVAnd ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/macros.jl:303
 [16] (::Distributed.var"#178#180"{UnitRange{Int64}, DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}})()
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/macros.jl:83
 [17] #invokelatest#2
    @ ./essentials.jl:729 [inlined]
 [18] invokelatest
    @ ./essentials.jl:726 [inlined]
 [19] #153
    @ ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:425 [inlined]
 [20] run_work_thunk(thunk::Distributed.var"#153#154"{Distributed.var"#178#180"{UnitRange{Int64}, DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}}, Tuple{}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, print_error::Bool)
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
 [21] run_work_thunk(rv::Distributed.RemoteValue, thunk::Function)
    @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:79
 [22] (::Distributed.var"#100#102"{Distributed.RemoteValue, Distributed.var"#153#154"{Distributed.var"#178#180"{UnitRange{Int64}, DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}}, Tuple{}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}})()
    @ Distributed ./task.jl:484
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:436
 [2] (::Distributed.var"#177#179"{DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}})()
   @ Distributed ./task.jl:455
ERROR: LoadError: TaskFailedException

    nested task error: SuiteSparse.CHOLMOD.CHOLMODException("out of memory")
    Stacktrace:
      [1] error_handler(status::Int32, file::Cstring, line::Int32, message::Cstring)
        @ SuiteSparse.CHOLMOD ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:127
      [2] cholmod_l_factorize_p(arg1::SuiteSparse.CHOLMOD.Sparse{Float64}, arg2::Base.RefValue{Float64}, arg3::Ptr{Nothing}, arg4::Int64, arg5::SuiteSparse.CHOLMOD.Factor{Float64}, arg6::Base.RefValue{SuiteSparse.LibSuiteSparse.cholmod_common_struct})
        @ SuiteSparse.LibSuiteSparse ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/lib/x86_64-linux-gnu.jl:1116
      [3] factorize_p!(A::SuiteSparse.CHOLMOD.Sparse{Float64}, β::Float64, F::SuiteSparse.CHOLMOD.Factor{Float64})
        @ SuiteSparse.CHOLMOD ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:616
      [4] cholesky!(F::SuiteSparse.CHOLMOD.Factor{Float64}, A::SuiteSparse.CHOLMOD.Sparse{Float64}; shift::Float64, check::Bool)
        @ SuiteSparse.CHOLMOD ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1147
      [5] #cholesky#8
        @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1185 [inlined]
      [6] cholesky
        @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1178 [inlined]
      [7] #cholesky#9
        @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1297 [inlined]
      [8] cholesky
        @ ~/julia-1.8.3/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:1297 [inlined]
      [9] factorize!(C::CovarIS{Float64, SparseArrays.SparseMatrixCSC{Float64, Int64}})
        @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/special_matrices.jl:117
     [10] DIVAnd_factorize!(s::DIVAnd.DIVAnd_struct{Float64, Int64, 3, SparseArrays.SparseMatrixCSC{Float64, Int64}})
        @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_factorize.jl:40
     [11] DIVAndrun(operatortype::Type, mask::BitArray{3}, pmnin::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, xiin::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, f::Vector{Float64}, lin::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}; velocity::Tuple{}, primal::Bool, factorize::Bool, tol::Float64, maxit::Int64, minit::Int64, constraints::Tuple{}, inversion::Symbol, moddim::Vector{Float64}, fracindex::Matrix{Float64}, alpha::Vector{Any}, keepLanczosVectors::Int64, compPC::typeof(DIVAnd_pc_none), progress::DIVAnd.var"#286#288", fi0::Array{Float64, 3}, f0::Vector{Float64}, alphabc::Float64, scale_len::Bool, btrunc::Vector{Any}, MEMTOFIT::Int64, topographyforfluxes::Tuple{}, fluxes::Tuple{}, epsfluxes::Int64, epsilon2forfractions::Int64, RTIMESONESCALES::Tuple{}, QCMETHOD::Tuple{}, coeff_laplacian::Vector{Float64}, coeff_derivative2::Vector{Float64}, mean_Labs::Vector{Float64})
        @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndrun.jl:125
     [12] #DIVAndrun#290
        @ ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndrun.jl:298 [inlined]
     [13] DIVAnd_cpme(mask::BitArray{3}, pmn::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, xi::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, f::Vector{Float64}, Labs::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}; csteps::Vector{Int64}, lmask::Vector{Any}, alphapc::Vector{Any}, mean_Labs::Vector{Float64}, otherargs::Base.Pairs{Symbol, Any, NTuple{6, Symbol}, NamedTuple{(:moddim, :MEMTOFIT, :QCMETHOD, :RTIMESONESCALES, :velocity, :coeff_derivative2), Tuple{Vector{Float64}, Int64, Tuple{}, Tuple{}, Tuple{}, Vector{Float64}}}})
        @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAnd_cpme.jl:88
     [14] macro expansion
        @ ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:281 [inlined]
     [15] (::DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool})(R::UnitRange{Int64}, lo::Int64, hi::Int64)
        @ DIVAnd ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/macros.jl:303
     [16] (::Distributed.var"#178#180"{UnitRange{Int64}, DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}})()
        @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/macros.jl:83
     [17] #invokelatest#2
        @ ./essentials.jl:729 [inlined]
     [18] invokelatest
        @ ./essentials.jl:726 [inlined]
     [19] #153
        @ ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:425 [inlined]
     [20] run_work_thunk(thunk::Distributed.var"#153#154"{Distributed.var"#178#180"{UnitRange{Int64}, DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}}, Tuple{}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, print_error::Bool)
        @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
     [21] run_work_thunk(rv::Distributed.RemoteValue, thunk::Function)
        @ Distributed ~/julia-1.8.3/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:79
     [22] (::Distributed.var"#100#102"{Distributed.RemoteValue, Distributed.var"#153#154"{Distributed.var"#178#180"{UnitRange{Int64}, DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}}, Tuple{}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}})()
        @ Distributed ./task.jl:484
    Stacktrace:
     [1] sync_end(c::Channel{Any})
       @ Base ./task.jl:436
     [2] (::Distributed.var"#177#179"{DIVAnd.var"#292#301"{3, Vector{Float64}, Int64, Tuple{}, Tuple{}, Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}}, BitArray{3}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, Vector{Float64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Symbol, Vector{Float64}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedVector{Float32}, SharedArrays.SharedArray{Float32, 3}, SharedArrays.SharedArray{Float32, 3}, Vector{Int64}, Vector{Float64}, Vector{Int64}, Vector{NTuple{6, Vector{Int64}}}, Bool, Bool}, UnitRange{Int64}})()
       @ Distributed ./task.jl:455
Stacktrace:
  [1] sync_end(c::Channel{Any})
    @ Base ./task.jl:436
  [2] macro expansion
    @ ./task.jl:455 [inlined]
  [3] DIVAndgo(mask::BitArray{3}, pmn::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, xi::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}}, f::Vector{Float64}, Labs::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}, errormethod::Symbol; moddim::Vector{Float64}, velocity::Tuple{}, MEMTOFIT::Int64, QCMETHOD::Tuple{}, RTIMESONESCALES::Tuple{}, solver::Symbol, overlapfactor::Float64, filteranom::Int64, filtererr::Int64, otherargs::Base.Pairs{Symbol, Vector{Float64}, Tuple{Symbol}, NamedTuple{(:coeff_derivative2,), Tuple{Vector{Float64}}}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/DIVAndgo.jl:115
  [4] (::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), DIVAnd.var"#639#642"{DIVAnd.var"#639#640#643"{TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Bool, NCDatasets.CFVariable{Union{Missing, Float32}, 4, NCDatasets.Variable{Float32, 4, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :missing_values, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Tuple{Float32}, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:solver, :MEMTOFIT, :coeff_derivative2), Tuple{Symbol, Int64, Vector{Float64}}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Vector{Float64}, Int64})(ds::NCDataset{Nothing})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/diva.jl:644
  [5] NCDataset(::DIVAnd.var"#447#463"{Vector{Float64}, OrderedDict{String, String}, OrderedDict{String, String}, typeof(DIVAnd.distfun_m), DIVAnd.var"#639#642"{DIVAnd.var"#639#640#643"{TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Bool, NCDatasets.CFVariable{Union{Missing, Float32}, 4, NCDatasets.Variable{Float32, 4, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :missing_values, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Tuple{Float32}, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, Int64, Bool, Bool, Dict{Any, Any}, Dict{Any, Any}, Int64, Float64, Int64, Float64, Float64, Tuple{}, Bool, Vector{Tuple{String, Float64}}, typeof(DIVAndgo), Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:solver, :MEMTOFIT, :coeff_derivative2), Tuple{Symbol, Int64, Vector{Float64}}}}, Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, Vector{Float64}, String, String, Bool, Matrix{DateTime}, Dict{Symbol, Any}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Vector{Float64}, Vector{Float64}, Int64, Tuple{Int64, Int64, Int64}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}, DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, Vector{DateTime}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Vector{Float64}, Int64}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/dataset.jl:241
  [6] NCDataset(::Function, ::String, ::Vararg{String})
    @ NCDatasets ~/.julia/packages/NCDatasets/h1epE/src/dataset.jl:238
  [7] diva3d(xi::Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Vector{Float64}, TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}}, x::Tuple{Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{DateTime}}, value::Vector{Float64}, len::Tuple{Array{Float64, 3}, Array{Float64, 3}, Array{Float64, 3}}, epsilon2::Vector{Float64}, filename::String, varname::String; datadir::String, bathname::String, bathisglobal::Bool, plotres::DIVAnd.var"#453#469", timeorigin::DateTime, moddim::Vector{Float64}, zlevel::Symbol, ncvarattrib::OrderedDict{String, String}, ncglobalattrib::OrderedDict{String, String}, transform::Tuple{DIVAnd.Anam.var"#trans#3"{DIVAnd.Anam.var"#trans#1#4"}, DIVAnd.Anam.var"#invtrans#5"{DIVAnd.Anam.var"#invtrans#2#6"}}, distfun::typeof(DIVAnd.distfun_m), mask::BitArray{3}, background::DIVAnd.var"#639#642"{DIVAnd.var"#639#640#643"{TimeSelectorYearListMonthList{Vector{UnitRange{Int64}}, Vector{UnitRange{Int64}}}, Bool, NCDatasets.CFVariable{Union{Missing, Float32}, 4, NCDatasets.Variable{Float32, 4, NCDataset{Nothing}}, NCDatasets.Attributes{NCDataset{Nothing}}, NamedTuple{(:fillvalue, :missing_values, :scale_factor, :add_offset, :calendar, :time_origin, :time_factor), Tuple{Float32, Tuple{Float32}, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}, background_epsilon2_factor::Nothing, background_lenz::Nothing, background_len::Nothing, background_lenz_factor::Int64, filterbackground::Int64, fitcorrlen::Bool, fithorzcorrlen::Bool, fitvertcorrlen::Bool, fithorz_param::Dict{Any, Any}, fitvert_param::Dict{Any, Any}, memtofit::Int64, overlapfactor::Float64, niter_e::Int64, minfield::Float64, maxfield::Float64, surfextend::Bool, velocity::Tuple{}, stat_per_timeslice::Bool, error_thresholds::Vector{Tuple{String, Float64}}, divamethod::typeof(DIVAndgo), kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:solver, :MEMTOFIT, :coeff_derivative2), Tuple{Symbol, Int64, Vector{Float64}}}})
    @ DIVAnd ~/.julia/packages/DIVAnd/MV3j9/src/diva.jl:375
  [8] macro expansion
    @ ./timing.jl:262 [inlined]
  [9] top-level scope
    @ ~/EMODnet-Chemistry/julia/european-climatologies/background.jl:395
 [10] include
    @ ./client.jl:476 [inlined]
 [11] top-level scope
    @ ~/EMODnet-Chemistry/julia/european-climatologies/run_clim.jl:31
jmbeckers commented 1 year ago

If the problem is too big and you have a shared memory system, there is little point to try using several processors at the same time I think, since they will compete for the same memory. Now if even a single process is asking for too much memory, you might try to reduce the memtofit parameter to activate also horizontal windowing with smaller problem sizes. Or try to shut down all tasks on your computer before launching the run. Under windows I also learned that it seems Julia is configured to at most use 75 percent of available memory.

Alexander-Barth commented 1 year ago

Indeed EOFError: read end of file, might be an out-of-memory error in disguise: has too little memory to do its job and the other process complains that it did not get all the data expected.

ctroupin commented 1 year ago

yes, in fact one of the reason I opened the issue is to document the error message, in case other users wonder why the read end of file.

Good thing is that some products have been created.

ctroupin commented 1 year ago

Closing the issue. So if somebody looks for Unhandled Task ERROR: EOFError: read end of file in the list of issues, they will see that it's probably related to memory.