search

Circuitscape / Omniscape.jl

Functions to compute omnidirectional landscape connectivity using circuit theory and the Omniscape algorithm.
https://docs.circuitscape.org/Omniscape.jl/stable/
MIT License
58 stars 12 forks source link

Omniscape fails on specific row / column: "median of empty array is undefined" #122

Open slamander opened 2 years ago

slamander commented 2 years ago

Cross-posting from gitter: there have two mentions of Omniscape failing on a specific row / column.

In my case, I receive this error: ERROR: ArgumentError: median of an empty array is undefined, Float64[] . The run will fail at different stages--sometimes quite far into a run, other times immediately--but always at the same row-column combination. I've tried to diagnose the error by changing the raster inputs or adjusting model parameters (e.g., solver, block size, r_cutoff [for source = resistance], and running conditional and non-conditional criteria).

Here's my error print-out. Let me know if additional information is needed. 

Progress:   9%|█████                                             |  ETA: 2:57:07
Omniscape failed on the moving window centered on row 325 column 5198
ERROR: TaskFailedException
Stacktrace:
 [1] wait
   @ .\task.jl:322 [inlined]
 [2] threading_run(func::Function)
   @ Base.Threads .\threadingconstructs.jl:34
 [3] macro expansion
   @ .\threadingconstructs.jl:93 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [6] top-level scope
   @ REPL[5]:1

    nested task error: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] macro expansion
       @ C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [3] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48

    caused by: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] median!(v::Vector{Float64})
       @ Statistics C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Statistics\src\Statistics.jl:791
     [2] median
       @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Statistics\src\Statistics.jl:836 [inlined]
     [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:231
     [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:172
     [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:308
     [6] macro expansion
       @ C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
     [7] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [8] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48
vlandau commented 1 year ago

Sorry for the delay in getting back to you on this. Is this still giving you problems?

It does make sense that the timing of the error being thrown is random since the order of the moving window solves is randomized for more accurate ETA estimates in the progress bar.

If possible, would you mind sharing your input files (and the associate ini file)? I'll try to take a look in the coming couple of weeks.

slamander commented 1 year ago

@vlandau, and my apologies for taking even longer to get back. Yes, this is still giving me issues, across several different input files. Here's the input data and ini file: https://drive.google.com/drive/folders/1kC4FVzw2HjlXVbLLot4a6LU0EPu-E-7s?usp=sharing.

slamander commented 1 year ago

By the way, here's the problem cells from two failed runs. The value of all of these cells are NA. And here's the stack trace from one of them: trouble

Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
 [6] top-level scope
   @ /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_All.jl:7

    nested task error: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [3] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30

    caused by: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] median!(v::Vector{Float64})
       @ Statistics /apps/julia/1.8.2/share/julia/stdlib/v1.8/Statistics/src/Statistics.jl:810
     [2] median
       @ /apps/julia/1.8.2/share/julia/stdlib/v1.8/Statistics/src/Statistics.jl:855 [inlined]
     [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:231
     [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:172
     [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:308
     [6] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
     [7] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [8] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [9] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_All.jl:7
Omniscape failed on the moving window centered on row 145 column 2049
Fri Mar 31 10:46:47 EDT 2023
Daviser95 commented 1 year ago

Hi guys, I'm having the exact same error launching a Europe-wide Omniscape scan. The error appeared after adding a conditional layer in the .ini file. I've already checked that the conditional layer has the same size and resolution as the resistance layer. I tried to change some parameters (radius and block size) but the error keeps appearing.

Omniscape failed on the moving window centered on row 3323 column 2228 ERROR: TaskFailedException

nested task error: ArgumentError: median of an empty array is undefined, Float64[]
Stacktrace:
 [1] macro expansion
   @ C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
 [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
   @ Omniscape .\threadingconstructs.jl:163
 [3] #161#threadsfor_fun
   @ .\threadingconstructs.jl:130 [inlined]
 [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
   @ Base.Threads .\threadingconstructs.jl:108

caused by: ArgumentError: median of an empty array is undefined, Float64[]
Stacktrace:
 [1] median!(v::Vector{Float64})
   @ Statistics C:\Users\utente\AppData\Local\Programs\Julia-1.9.0\share\julia\stdlib\v1.9\Statistics\src\Statistics.jl:803
 [2] median
   @ C:\Users\utente\AppData\Local\Programs\Julia-1.9.0\share\julia\stdlib\v1.9\Statistics\src\Statistics.jl:848 [inlined]
 [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:219
 [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:172
 [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:308
 [6] macro expansion
   @ C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
 [7] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
   @ Omniscape .\threadingconstructs.jl:163
 [8] #161#threadsfor_fun
   @ .\threadingconstructs.jl:130 [inlined]
 [9] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
   @ Base.Threads .\threadingconstructs.jl:108

Stacktrace: [1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool) @ Base.Threads .\threadingconstructs.jl:120 [2] macro expansion @ .\threadingconstructs.jl:168 [inlined] [3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool) @ Omniscape C:\Users\utente.julia\packages\Omniscape\9gHf2\src\main.jl:257 [4] run_omniscape(path::String) @ Omniscape C:\Users\utente.julia\packages\Omniscape\9gHf2\src\main.jl:536 [5] top-level scope @ REPL[6]:1

vlandau commented 1 year ago

My best guess is at the column/row of one of your inputs, the resulting slice/chunks of the array is just empty (all NaN values). Would you be able to clip out a section (square, with side lengths equal to 2×radius + 1) of each of your input rasters centered on the failing row/column and share it with me?

Daviser95 commented 1 year ago

Hi @vlandau and thanks for replying, you're probably right. I share the clipped input rasters with your options. However, if this is the problem, I don't know why using only the resistance raster doesn't cause problems, while this error only occurs when using a second layer as a condition. Moreover, I noticed that the resolution of the two layers is not the same, while the coordinate system and the dimensions are, but I don't know if this difference can cause the error.

Immagine 2023-06-03 030331 Immagine 2023-06-03 030516

ashton314 commented 1 year ago

Hi there! I'm a PhD student and I'm building a tool to help developers automatically track down where NaNs come from in computations, and I'd like to try it out on the problem you're having here to see if I can help. Would someone mind showing me how to fire it all off to reproduce the problems that you're having here? I think I have the .ini and the .tiff files already—just wondering how to set it up properly.

Daviser95 commented 1 year ago

Hi @ashton314! If you already have .ini and .tiff file, I think you just need to donwload Omniscape on Julia and then run the .ini file. You can get information on this page https://docs.circuitscape.org/Omniscape.jl/stable/usage/#Running-Omniscape

slamander commented 1 year ago

@vlandau I also thought there could be NaNs on the failing pixels, but after investigating the failures I found no NaNs. Does it make sense that identical jobs fail at different locations/durations? Sometimes the job will near completion (>90%), but others will fail early--all failing on different pixels. Unlike @Daviser95, some jobs will fail whether or not I have conditional criteria (it's not very predictable). Here's an output from the last job I ran without conditional criteria:

Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
 [6] top-level scope
   @ /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_Americas.jl:7

    nested task error: 
Progress:  56%|████████████████████████████▏                     |  ETA: 6:19:09
Progress:  56%|████████████████████████████▏                     |  ETA: 6:19:09AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [3] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30

    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SuiteSparse.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/core.jl:621
      [2] macro expansion
        @ ./timing.jl:382 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/raster/advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/raster/advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:332
      [7] macro expansion
        @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape ./threadingconstructs.jl:84
      [9] #161#threadsfor_fun
        @ ./threadingconstructs.jl:51 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_Americas.jl:7

Progress:  56%|████████████████████████████▏                     |  ETA: 6:19:08Omniscape failed on the moving window centered on row 2542 column 7157
Omniscape failed on the moving window centered on row 1424 column 1983
Omniscape failed on the moving window centered on row 1593 column 4102
Wed Jul 12 21:26:43 EDT 2023
vlandau commented 1 year ago

It does make sense that it would occur at different times because the order in which the windows are solved is random (this helps the ETA on the progress bar to be more accurate). I assume in your case there are multiple moving window locations that will cause failure, but which one you run into first is random due to the random solve order.

geomenke commented 8 months ago

I regularly get the same issue as @slamander and @Daviser95. Omniscape will fail on specific row / columns. These are not No Data or Nan pixels however. I have clipped out a small area centered on these pixels and run OS on the subset and it finishes successfully. What are the potential reasons for this type of error? Here is the stacktrace of the error. 

moving window centered on row 6461 column 8703
Progress:  92%|███████████████████████████████████████████████   |  ETA: 4:55:16ERROR: TaskFailedException

    nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:194
     [3] #161#threadsfor_fun
       @ .\threadingconstructs.jl:161 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads .\threadingconstructs.jl:139

    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SparseArrays.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\core.jl:621
      [2] macro expansion
        @ .\timing.jl:393 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\utils.jl:332
      [7] macro expansion
        @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape .\threadingconstructs.jl:194
      [9] #161#threadsfor_fun
        @ .\threadingconstructs.jl:161 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads .\threadingconstructs.jl:139
Stacktrace:
 [1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads .\threadingconstructs.jl:151
 [2] macro expansion
   @ .\threadingconstructs.jl:199 [inlined]
 [3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [4] run_omniscape(path::String)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [5] top-level scope
   @ REPL[3]:1
vlandau commented 8 months ago

@geomenke this is a different error than the empty array error mentioned above (but I see now one of the stack traces had an AssertionError too), coming directly from Circuitscape, but fortunately, we just made a patch release to Circuitscape that may fix that issue. Would you update to Circuitscape 5.13.2 (you'll need Julia 1.9) and try again?

geomenke commented 8 months ago

@vlandau I am already at Julia 1.9.2 so what is the best way to update to Circuitscape 5.13.2? Do I upgrade Omniscape? Can that be done via the command line?

vlandau commented 8 months ago

Oh great! To update, in a Julia terminal, enter the package interface by typing the ] key, then run update Circuitscape and you should see 5.13.2 get installed.

geomenke commented 8 months ago

It did not specifically list Circuitscape as being updated, just things like updating registry etc. How can I check which Circuitscape version I have?

vlandau commented 8 months ago

In the same package prompt/interface you should be able to run status Circuitscape

geomenke commented 8 months ago

I see that I have an outdated and restricted by compatibility contraints version of Omniscape and that Circuitscape did not update after running update. Running update Omniscape did not update Omniscape. Do I need to run using Pkg; Pkg.add(PackageSpec(name = "Omniscape", rev = "main)) image

vlandau commented 8 months ago

You shouldn't need to do that. There are some solutions in this thread that might work for you: https://discourse.julialang.org/t/how-to-force-an-update/73071/12

geomenke commented 8 months ago

I updated to Circuitscape v5.13.2 and Omniscape v0.6.1, but it failed again with a similar stacktrace as before: Any recommendations?

Progress:  81%|█████████████████████████████████████████         |  ETA: 17:26:09Omniscape failed on the moving window centered on row 11263 column 10839
Progress:  97%|█████████████████████████████████████████████████ |  ETA: 2:58:35ERROR: TaskFailedException

    nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:194
     [3] #161#threadsfor_fun
       @ .\threadingconstructs.jl:161 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads .\threadingconstructs.jl:139

    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SparseArrays.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\core.jl:621
      [2] macro expansion
        @ .\timing.jl:393 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\utils.jl:332
      [7] macro expansion
        @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape .\threadingconstructs.jl:194
      [9] #161#threadsfor_fun
        @ .\threadingconstructs.jl:161 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads .\threadingconstructs.jl:139
Stacktrace:
 [1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads .\threadingconstructs.jl:151
 [2] macro expansion
   @ .\threadingconstructs.jl:199 [inlined]
 [3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [4] run_omniscape(path::String)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [5] top-level scope
   @ REPL[14]:1
ptfreeman-csp commented 2 months ago

@slamander or @vlandau - we are also running into this issue using conditional files to incorporate climate connectivity into our models. We've exported all of our input layers at the same resolution and CRS but we're running into the same error with the latest version of Omniscape. Was this issue ever resolved?

vlandau commented 1 month ago

@ptfreeman-csp hmm, no I'm not sure why this is happening, but I guess either something is wrong with the input, or Omniscape's processing of the input. Do you have an exact column and row that it failed on? Taking a look around that area/coordinate of all of the inputs might shed some light about what could be going on.

ptfreeman-csp commented 1 month ago

I think we figured out what the issue was. I had too many threads going and it was burning through the memory of our VM and unable to actually complete the solving calculations. We ultimately had to reduce our moving window and block sizes and reduce the total number of threads to stay under the necessary memory limit. Updating to the latest version of Julia and Omniscape also seemed to help.

On Mon, Aug 26, 2024 at 5:57 PM Vincent Landau @.***> wrote:

@ptfreeman-csp https://github.com/ptfreeman-csp hmm, no I'm not sure why this is happening, but I guess either something is wrong with the input, or Omniscape's processing of the input. Do you have an exact column and row that it failed on? Taking a look around that area/coordinate of all of the inputs might shed some light about what could be going on.

— Reply to this email directly, view it on GitHub https://github.com/Circuitscape/Omniscape.jl/issues/122#issuecomment-2311367624, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUE56Z47SM34O5JY37E4JELZTPFGXAVCNFSM6AAAAAAQY5S6T2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGM3DONRSGQ . You are receiving this because you were mentioned.Message ID: @.***>

ptfreeman-csp commented 1 month ago

Also - for whatever reason we had to only allow the values of the conditions to be rounded to the nearest tenth. Anything that had two decimal points in the conditions refused to run but as soon as we knocked it down to a single decimal point then Omniscape decided to cooperate.

On Mon, Aug 26, 2024 at 6:10 PM Patrick Freeman @.***> wrote:

I think we figured out what the issue was. I had too many threads going and it was burning through the memory of our VM and unable to actually complete the solving calculations. We ultimately had to reduce our moving window and block sizes and reduce the total number of threads to stay under the necessary memory limit. Updating to the latest version of Julia and Omniscape also seemed to help.

On Mon, Aug 26, 2024 at 5:57 PM Vincent Landau @.***> wrote:

@ptfreeman-csp https://github.com/ptfreeman-csp hmm, no I'm not sure why this is happening, but I guess either something is wrong with the input, or Omniscape's processing of the input. Do you have an exact column and row that it failed on? Taking a look around that area/coordinate of all of the inputs might shed some light about what could be going on.

— Reply to this email directly, view it on GitHub https://github.com/Circuitscape/Omniscape.jl/issues/122#issuecomment-2311367624, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUE56Z47SM34O5JY37E4JELZTPFGXAVCNFSM6AAAAAAQY5S6T2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGM3DONRSGQ . You are receiving this because you were mentioned.Message ID: @.***>

vlandau commented 1 month ago

Oh that second point is very strange. Glad it's working now though!