Closed mir123 closed 1 year ago
vlandau
commented
1 year ago @mir123 This due to Circuitscape having some solution errors over the acceptable tolerance. Is this the full stacktrace? Omniscape should be printing information on the row and column at which the failure occurred when an error is thrown. Something about the source and resistance within a specific moving window solve is giving Circuitscape a hard time.
cc @ranjanan
mir123
commented
1 year ago I did get the following errors before:
Progress: 41%|████████████████████▍ | ETA: 1 days, 9:05:50Omniscape failed on the moving window centered on row 4310 column 1879
Progress: 42%|████████████████████▉ | ETA: 1 days, 8:32:22Omniscape failed on the moving window centered on row 6520 column 5636
Progress: 53%|██████████████████████████▍ | ETA: 1 days, 1:34:06Omniscape failed on the moving window centered on row 3647 column 17570
Progress: 54%|███████████████████████████▎ | ETA: 1 days, 0:29:07Omniscape failed on the moving window centered on row 2542 column 13371
Progress: 74%|█████████████████████████████████████▏ | ETA: 13:42:28Omniscape failed on the moving window centered on row 5194 column 19780
Progress: 76%|█████████████████████████████████████▉ | ETA: 12:56:28Omniscape failed on the moving window centered on row 2763 column 13371
Progress: 91%|█████████████████████████████████████████████▋ | ETA: 4:41:19ERROR: TaskFailedExceptionAfter the TaskFailedException I got the rest of the error above.
Plotting the points shows they end up all in a line (here with resistance surface as background, yellow circle is radius)
ptfreeman-csp
commented
1 year ago I am having a similar issue (I think) -- a similar error message ( nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6)
Julia version: Version 1.6.1 Omniscape version: v0.5.8
Config Info
block_size = 35
radius = 1066
source_threshold = 0.03556
calc_normalized_current = false
solver = cholmod
parallelize = true
parallel_batch_size = 10Omniscape failed on the moving window centered on row 5653 column 6633
Omniscape failed on the moving window centered on row 3273 column 5618
Omniscape failed on the moving window centered on row 4988 column 6528
Omniscape failed on the moving window centered on row 928 column 2923
Omniscape failed on the moving window centered on row 4848 column 1873
Omniscape failed on the moving window centered on row 4218 column 438
Omniscape failed on the moving window centered on row 998 column 1243
Omniscape failed on the moving window centered on row 4953 column 928
Omniscape failed on the moving window centered on row 6913 column 5548
Omniscape failed on the moving window centered on row 1138 column 2853
ERROR: Omniscape failed on the moving window centered on row 5583 column 3658TaskFailedException
Omniscape failed on the moving window centered on row 4568 column 2013
Omniscape failed on the moving window centered on row 1873 column 3483
Stacktrace:
Omniscape failed on the moving window centered on row 2783 column 1838
Omniscape failed on the moving window centered on row 1453 column 2678
Omniscape failed on the moving window centered on row 2258 column 2258
[1] wait
@ ./task.jl:322 [inlined]
[2] threading_run(func::Function)
@ Base.Threads ./threadingconstructs.jl:34
[3] macro expansion
@ ./threadingconstructs.jl:93 [inlined]
[4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
@ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
[5] run_omniscape(path::String)
@ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
[6] top-level scope
@ REPL[1]:1
nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
Stacktrace:
[1] macro expansion
@ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
[2] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
@ Omniscape ./threadingconstructs.jl:81
[3] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
@ Omniscape ./threadingconstructs.jl:48
caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
Stacktrace:
[1] solve_linear_system(factor::SuiteSparse.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/core.jl:635
[2] macro expansion
@ ./timing.jl:287 [inlined]
[3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/raster/advanced.jl:321
[4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/raster/advanced.jl:292
[5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/utils.jl:564
[6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
@ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:332
[7] macro expansion
@ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
[8] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
@ Omniscape ./threadingconstructs.jl:81
[9] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
@ Omniscape ./threadingconstructs.jl:48
Unlike mir123, when I look at the location of the error-causing cells there doesn't appear to be any clear spatial pattern other than perhaps some moving windows containing a lot of no-data cells?
vlandau
commented
1 year ago @ptfreeman-csp unfortunately this is indeed a problem with Circuitscape. From what I'm seeing, it does seem to happen more when people are using large moving windows. You might consider cross posting this to the Circuitscape repo as an issue over there?
ptfreeman-csp
commented
1 year ago @vlandau - ah well shoot. Thanks for taking the time to reply! I will look into cross posting this to Circuitscape and/or trying to reduce my moving window size to see if that ameliorates the issue.
vlandau
commented
1 year ago I'll add, this also isn't necessarily an issue with Circuitscape, but rather the linear solvers it's using. It's strange that you're getting these errors when using cholmod because that's supposed to be a direct brute-force solver.
ptfreeman-csp
commented
1 year ago Cross-posting back here that I submitted an issue to Circuitscape here. I will say that I updated to the latest versions of Omniscape and Circuitscape and did get a slightly different error message but I don't have the wherewithal to understand it. Reposting here in case there's anything obvious I'm missing:
Stacktrace:
[1] wait
@ ./task.jl:322 [inlined]
[2] threading_run(func::Function)
@ Base.Threads ./threadingconstructs.jl:34
[3] macro expansion
@ ./threadingconstructs.jl:93 [inlined]
[4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
@ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
[5] run_omniscape(path::String)
@ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
[6] top-level scope
@ REPL[5]:1
nested task error: AssertionError: norm(G * v .- curr) / norm(curr) < 1.0e-6
Stacktrace:
[1] macro expansion
@ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
[2] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
@ Omniscape ./threadingconstructs.jl:81
[3] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
@ Omniscape ./threadingconstructs.jl:48
caused by: AssertionError: norm(G * v .- curr) / norm(curr) < 1.0e-6
Stacktrace:
[1] solve_linear_system(G::SparseArrays.SparseMatrixCSC{Float64, Int64}, curr::Vector{Float64}, M::AlgebraicMultigrid.Preconditioner{AlgebraicMultigrid.MultiLevel{AlgebraicMultigrid.Pinv{Float64}, AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, SparseArrays.SparseMatrixCSC{Float64, Int64}, SparseArrays.SparseMatrixCSC{Float64, Int64}, LinearAlgebra.Adjoint{Float64, SparseArrays.SparseMatrixCSC{Float64, Int64}}, AlgebraicMultigrid.MultiLevelWorkspace{Vector{Float64}, 1}}, AlgebraicMultigrid.V})
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/core.jl:616
[2] macro expansion
@ ./timing.jl:287 [inlined]
[3] multiple_solve(s::Circuitscape.AMGSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/raster/advanced.jl:312
[4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.AMGSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/raster/advanced.jl:292
[5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
@ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/utils.jl:564
[6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
@ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:332
[7] macro expansion
@ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
[8] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
@ Omniscape ./threadingconstructs.jl:81
[9] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
@ Omniscape ./threadingconstructs.jl:48All inputs are here. I can give you access as needed.
ptfreeman-csp
commented
1 year ago Another update on this and hoping that this might help someone ID what the source of the problem is. I examined the values in my source and resistance layers at the row/column combinations that are throwing the "moving window failed at" messages and the vast majority (but not all of them) are occurring where source values are NaN -- could this be a potential problem? I thought that these no-data/nan values were set to zero? And for what's it's worth -- almost all of these "failure points" are occurring in areas of relatively high resistance in our resistance surfaces (which in this case almost exclusively correspond to areas that are roads)
ID source_value resistance_value
1 1 NaN 2076.2005
2 2 NaN 11042.0864
3 3 NaN 8652.3815
4 4 NaN 8732.2469
5 5 0.21210049 1089.3377
6 6 NaN 2783.3380
7 7 0.15325072 1039.4780
8 8 NaN 8953.6835
9 9 0.09967852 9022.2103
10 10 0.27039680 1426.6000
11 11 0.29848063 1096.9960
12 12 NaN 8813.7933
13 13 NaN 9282.7749
14 14 0.40513211 705.9667
15 15 NaN 2374.9736
16 16 NaN 8972.9894
17 17 NaN 8837.5093
18 18 NaN 1008.4568
19 19 NaN 1205.6870
20 20 NaN 2000.0000
21 21 NaN 1259.6216
22 22 NaN 3874.8210
vlandau
commented
1 year ago Good find @ptfreeman-csp. Thanks for taking a deeper look. I'll try to take a look this weekend. You may be onto something here.
vlandau
commented
1 year ago Good find @ptfreeman-csp. Thanks for taking a deeper look. I'll try to take a look this weekend. You may be onto something here.
gagecarto
commented
1 year ago Hey @vlandau & @ptfreeman-csp - Funny seeing you both chatting about this recently.
I am re-doing some big runs in Omniscape and getting similar errors. I am going to keep digging but figure I'll post the trace here in case it's helpful
Progress: 66%|█████████████████████████████████▏ | ETA: 1:51:36Omniscape failed on the moving window centered on row 653 column 5666 TaskFailedExceptionProgress: 74%|█████████████████████████████████████ | ETA: 1:27:36 Stacktrace: [1] wait @ ./task.jl:345 [inlined] [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool) @ Base.Threads ./threadingconstructs.jl:38 [3] macro expansion @ ./threadingconstructs.jl:89 [inlined] [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool) @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257 [5] run_omniscape(path::String) @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536 [6] top-level scope @ REPL[6]:1 nested task error: AssertionError: norm(G * v .- curr) / norm(curr) < 1.0e-6 Stacktrace: [1] macro expansion @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined] [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool) @ Omniscape ./threadingconstructs.jl:84 [3] #161#threadsfor_fun @ ./threadingconstructs.jl:51 [inlined] [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})() @ Base.Threads ./threadingconstructs.jl:30 caused by: AssertionError: norm(G * v .- curr) / norm(curr) < 1.0e-6 Stacktrace: [1] solve_linear_system(G::SparseArrays.SparseMatrixCSC{Float64, Int64}, curr::Vector{Float64}, M::AlgebraicMultigrid.Preconditioner{AlgebraicMultigrid.MultiLevel{AlgebraicMultigrid.Pinv{Float64}, AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, SparseArrays.SparseMatrixCSC{Float64, Int64}, SparseArrays.SparseMatrixCSC{Float64, Int64}, LinearAlgebra.Adjoint{Float64, SparseArrays.SparseMatrixCSC{Float64, Int64}}, AlgebraicMultigrid.MultiLevelWorkspace{Vector{Float64}, 1}}, AlgebraicMultigrid.V}) @ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/core.jl:616 [2] macro expansion @ ./timing.jl:382 [inlined] [3] multiple_solve(s::Circuitscape.AMGSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool) @ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/raster/advanced.jl:312 [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.AMGSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64}) @ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/raster/advanced.jl:292 [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String}) @ Circuitscape ~/.julia/packages/Circuitscape/XpftG/src/utils.jl:564 [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType) @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:332 [7] macro expansion @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined] [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool) @ Omniscape ./threadingconstructs.jl:84 [9] #161#threadsfor_fun @ ./threadingconstructs.jl:51 [inlined] [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})() @ Base.Threads ./threadingconstructs.jl:30
ptfreeman-csp
commented
1 year ago @gagecarto yup -- that is basically the same exact error I'm getting (I think) and it's an issue with the solver employed by Circuitscape but I don't know how to "hack it" to have a higher error tolerance. I am experimenting with shrinking my moving window sizes to see if/where I manage to get a successful run but it's been a bit of a shot in the dark. If @ranjanan and @vlandau have any suggestions on how to overcome the error in the solver component I would be thrilled. @gagecarto do your source inputs have a lot of missing/masked areas (e.g. over things like urban areas) that also have corresponding relatively high resistance values? Just curious if that's a common factor here... In general I also only get the crash right before the Omniscape run is done :cry:
gagecarto
commented
1 year ago @ptfreeman-csp - Yep, our highly resistant areas are not null, but have been given high values
class : SpatRaster dimensions : 7092, 8628, 1 (nrow, ncol, nlyr) resolution : 1000, 1000 (x, y) extent : -5439267, 3188733, -893546.7, 6198453 (xmin, xmax, ymin, ymax) min value : 1.00 max value : 98999.55
Now that I am further reviewing our input data, the ocean should be null, not high. We do want to leave our lakes and rivers as high but removing all those unnecessary ocean pixels will be my next step before a rerun
ptfreeman-csp
commented
1 year ago @gagecarto We seem to have generally the same idea -- although I will say that I have plenty of areas that are "no-data" - we're working in the desert in southern California but our buffer area does extend out to no-data ocean and I've still been running into issues. Have you plotted the particular cells that Omniscape is failing at? Out of curiosity are the 'no-data' areas in your raster labeled as true NAs or are they NaN?
gagecarto
commented
1 year ago @ptfreeman-csp - I've not plotted the problem cells yet. My no data values are -3.4028234663852886e+38
Size is 8628, 7092 Coordinate System is: PROJCRS["USA_Contiguous_Albers_Equal_Area_Conic_USGS_version", BASEGEOGCRS["NAD83", DATUM["North American Datum 1983", ELLIPSOID["GRS 1980",6378137,298.257222101004, LENGTHUNIT["metre",1]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4269]], CONVERSION["Albers Equal Area", METHOD["Albers Equal Area", ID["EPSG",9822]], PARAMETER["Latitude of false origin",23, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8821]], PARAMETER["Longitude of false origin",-96, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8822]], PARAMETER["Latitude of 1st standard parallel",29.5, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8823]], PARAMETER["Latitude of 2nd standard parallel",45.5, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8824]], PARAMETER["Easting at false origin",0, LENGTHUNIT["metre",1], ID["EPSG",8826]], PARAMETER["Northing at false origin",0, LENGTHUNIT["metre",1], ID["EPSG",8827]]], CS[Cartesian,2], AXIS["easting",east, ORDER[1], LENGTHUNIT["metre",1, ID["EPSG",9001]]], AXIS["northing",north, ORDER[2], LENGTHUNIT["metre",1, ID["EPSG",9001]]]] Data axis to CRS axis mapping: 1,2 Origin = (-5439266.590459235012531,6198453.271350046619773) Pixel Size = (1000.000000000000000,-1000.000000000000000) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left (-5439266.590, 6198453.271) (171d51' 8.96"E, 53d 5'39.03"N) Lower Left (-5439266.590, -893546.729) (140d15'30.90"W, 1d49'52.56"N) Upper Right ( 3188733.410, 6198453.271) ( 28d47'12.17"W, 70d33'47.72"N) Lower Right ( 3188733.410, -893546.729) ( 68d46'11.86"W, 10d10'58.14"N) Center (-1125266.590, 2652453.271) (110d34'51.02"W, 46d 4'55.16"N) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray Min=1.000 Max=98999.547 Minimum=1.000, Maximum=98999.547, Mean=40603.498, StdDev=42833.884 NoData Value=-3.4028234663852886e+38 Metadata: STATISTICS_MAXIMUM=98999.546875 STATISTICS_MEAN=40603.497584455 STATISTICS_MINIMUM=1 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=42833.883683251
ptfreeman-csp
commented
1 year ago @gagecarto I have no idea if this is a potential issue or not but I am wondering if there's something funky going on with the no-data values. Can I ask how you generated your source and resistance layers (i.e. in R, GEE, something else?)?
gagecarto
commented
1 year ago @ptfreeman-csp - Rasters were generated and export using Terrra in R. I've now had some successful runs with smaller subsets of the data. I am slowly increasing the extent and adjusting parameters, hoping to find out what's crashing the bigger runs.
mir123
commented
1 year ago I generated my layers using the gdal raster calculator.
gagecarto
commented
1 year ago @ptfreeman-csp @mir123 @vlandau - I struggled with these errors for many days. I then remembered I had as saved EC2 image from previous Omniscape runs in 2021. I fired this image up which was running Julia 1.6.2, Omniscape and other dependencies installed at that time. Everything is now working how it should. I pulled all the package folders etc if anyone wants a copy.
ptfreeman-csp
commented
1 year ago I was able to re-create the environment/project from @gagecarto using his project.toml and manifest.toml files and I also got everything to run properly.
coport
commented
1 year ago @ptfreeman-csp @mir123 @vlandau - I struggled with these errors for many days. I then remembered I had as saved EC2 image from previous Omniscape runs in 2021. I fired this image up which was running Julia 1.6.2, Omniscape and other dependencies installed at that time. Everything is now working how it should. I pulled all the package folders etc if anyone wants a copy.
Hi @gagecarto I am having a similar issue and would love the files to recreate the older environment if possible!
gagecarto
commented
1 year ago Hi @coport - below is the contents of my Manifest.toml and Project.toml I believe @ptfreeman-csp was able to use these to install functioning packages etc and complete the analysis
PROJECT.TOML
[deps]
Omniscape = "a38d70fc-99f5-11e9-1e3d-cbca093024c3"MANIFEST.TOML
# This file is machine-generated - editing it directly is not advised
[[AlgebraicMultigrid]]
deps = ["CompatHelper", "DelimitedFiles", "IterativeSolvers", "LinearAlgebra", "Printf", "Random", "SparseArrays", "Test"]
git-tree-sha1 = "4d23bc92a192f49206cb96478fdd81c03f8f77ca"
uuid = "2169fc97-5a83-5252-b627-83903c6c433c"
version = "0.4.0"
[[ArchGDAL]]
deps = ["Dates", "DiskArrays", "GDAL", "GeoFormatTypes", "GeoInterface", "Tables"]
git-tree-sha1 = "c5196140e54514a869d012332f3ab0b02ad2b337"
uuid = "c9ce4bd3-c3d5-55b8-8973-c0e20141b8c3"
version = "0.5.3"
[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"
[[ArnoldiMethod]]
deps = ["LinearAlgebra", "Random", "StaticArrays"]
git-tree-sha1 = "f87e559f87a45bece9c9ed97458d3afe98b1ebb9"
uuid = "ec485272-7323-5ecc-a04f-4719b315124d"
version = "0.1.0"
[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"
[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"
[[Circuitscape]]
deps = ["AlgebraicMultigrid", "ArchGDAL", "Dates", "DelimitedFiles", "Distributed", "GZip", "IterativeSolvers", "LightGraphs", "LinearAlgebra", "Logging", "Pardiso", "REPL", "SharedArrays", "SimpleWeightedGraphs", "SparseArrays", "SuiteSparse", "Test"]
git-tree-sha1 = "b55ede0dc66843c35b17f5eb8665303c70fa7f9d"
uuid = "2b7a1792-8151-5239-925d-e2b8fdfa3201"
version = "5.8.3"
[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "727e463cfebd0c7b999bbf3e9e7e16f254b94193"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.34.0"
[[CompatHelper]]
deps = ["Base64", "Dates", "GitHub", "HTTP", "JSON", "Pkg", "Printf", "TOML", "TimeZones", "UUIDs"]
git-tree-sha1 = "d2dabdd67f4599bdfabdb4070ee367f70107bbe7"
uuid = "aa819f21-2bde-4658-8897-bab36330d9b7"
version = "1.18.6"
[[DataAPI]]
git-tree-sha1 = "ee400abb2298bd13bfc3df1c412ed228061a2385"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.7.0"
[[DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "7d9d316f04214f7efdbb6398d545446e246eff02"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.10"
[[DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"
[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"
[[DiskArrays]]
git-tree-sha1 = "599dc32bae654fa78056b15fed9b2af36f04ee44"
uuid = "3c3547ce-8d99-4f5e-a174-61eb10b00ae3"
version = "0.2.11"
[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"
[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
[[Expat_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b3bfd02e98aedfa5cf885665493c5598c350cd2f"
uuid = "2e619515-83b5-522b-bb60-26c02a35a201"
version = "2.2.10+0"
[[ExprTools]]
git-tree-sha1 = "b7e3d17636b348f005f11040025ae8c6f645fe92"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.6"
[[Future]]
deps = ["Random"]
uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820"
[[GDAL]]
deps = ["CEnum", "GDAL_jll", "NetworkOptions", "PROJ_jll"]
git-tree-sha1 = "8b1df30f3b390ab1ce8329a5fbf7c3a2f85ee5a9"
uuid = "add2ef01-049f-52c4-9ee2-e494f65e021a"
version = "1.2.3"
[[GDAL_jll]]
deps = ["Artifacts", "Expat_jll", "GEOS_jll", "JLLWrappers", "LibCURL_jll", "LibSSH2_jll", "Libdl", "Libtiff_jll", "MbedTLS_jll", "OpenJpeg_jll", "PROJ_jll", "Pkg", "SQLite_jll", "Zlib_jll", "Zstd_jll", "libgeotiff_jll", "nghttp2_jll"]
git-tree-sha1 = "439c33eb4dfa74a43a1e96b1d758aeb3cbc33dc3"
uuid = "a7073274-a066-55f0-b90d-d619367d196c"
version = "300.202.100+0"
[[GEOS_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "45d0ddfd29620ac9b2d1072801e90fb016c5f94c"
uuid = "d604d12d-fa86-5845-992e-78dc15976526"
version = "3.9.0+0"
[[GZip]]
deps = ["Libdl"]
git-tree-sha1 = "039be665faf0b8ae36e089cd694233f5dee3f7d6"
uuid = "92fee26a-97fe-5a0c-ad85-20a5f3185b63"
version = "0.5.1"
[[GeoFormatTypes]]
git-tree-sha1 = "bb75ce99c9d6fb2edd8ef8ee474991cdacf12221"
uuid = "68eda718-8dee-11e9-39e7-89f7f65f511f"
version = "0.3.0"
[[GeoInterface]]
deps = ["RecipesBase"]
git-tree-sha1 = "38a649e6a52d1bea9844b382343630ac754c931c"
uuid = "cf35fbd7-0cd7-5166-be24-54bfbe79505f"
version = "0.5.5"
[[GitHub]]
deps = ["Base64", "Dates", "HTTP", "JSON", "MbedTLS", "Sockets", "SodiumSeal"]
git-tree-sha1 = "9e62f3eae4a2514d1de65cc2ac9d1669032a4595"
uuid = "bc5e4493-9b4d-5f90-b8aa-2b2bcaad7a26"
version = "5.6.0"
[[HTTP]]
deps = ["Base64", "Dates", "IniFile", "Logging", "MbedTLS", "NetworkOptions", "Sockets", "URIs"]
git-tree-sha1 = "44e3b40da000eab4ccb1aecdc4801c040026aeb5"
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
version = "0.9.13"
[[Inflate]]
git-tree-sha1 = "f5fc07d4e706b84f72d54eedcc1c13d92fb0871c"
uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9"
version = "0.1.2"
[[IniFile]]
deps = ["Test"]
git-tree-sha1 = "098e4d2c533924c921f9f9847274f2ad89e018b8"
uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f"
version = "0.5.0"
[[IntelOpenMP_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d979e54b71da82f3a65b62553da4fc3d18c9004c"
uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0"
version = "2018.0.3+2"
[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
[[IterativeSolvers]]
deps = ["LinearAlgebra", "Printf", "Random", "RecipesBase", "SparseArrays"]
git-tree-sha1 = "1a8c6237e78b714e901e406c096fc8a65528af7d"
uuid = "42fd0dbc-a981-5370-80f2-aaf504508153"
version = "0.9.1"
[[IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"
[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"
[[JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "8076680b162ada2a031f707ac7b4953e30667a37"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.2"
[[JpegTurbo_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d735490ac75c5cb9f1b00d8b5509c11984dc6943"
uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8"
version = "2.1.0+0"
[[LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"
[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"
[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"
[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"
[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"
[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
[[Libtiff_jll]]
deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"]
git-tree-sha1 = "340e257aada13f95f98ee352d316c3bed37c8ab9"
uuid = "89763e89-9b03-5906-acba-b20f662cd828"
version = "4.3.0+0"
[[LightGraphs]]
deps = ["ArnoldiMethod", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"]
git-tree-sha1 = "432428df5f360964040ed60418dd5601ecd240b6"
uuid = "093fc24a-ae57-5d10-9952-331d41423f4d"
version = "1.3.5"
[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
[[LittleCMS_jll]]
deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg"]
git-tree-sha1 = "110897e7db2d6836be22c18bffd9422218ee6284"
uuid = "d3a379c0-f9a3-5b72-a4c0-6bf4d2e8af0f"
version = "2.12.0+0"
[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"
[[MKL_jll]]
deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"]
git-tree-sha1 = "c253236b0ed414624b083e6b72bfe891fbd2c7af"
uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7"
version = "2021.1.1+1"
[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "0fb723cd8c45858c22169b2e42269e53271a6df7"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.7"
[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"
[[MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.3"
[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
[[Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "2ca267b08821e86c5ef4376cffed98a46c2cb205"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.0.1"
[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"
[[Mocking]]
deps = ["ExprTools"]
git-tree-sha1 = "748f6e1e4de814b101911e64cc12d83a6af66782"
uuid = "78c3b35d-d492-501b-9361-3d52fe80e533"
version = "0.7.2"
[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"
[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"
[[Omniscape]]
deps = ["ArchGDAL", "Circuitscape", "DelimitedFiles", "LinearAlgebra", "ProgressMeter", "Random", "Statistics", "StatsBase", "Test"]
git-tree-sha1 = "63fda64982fb61c3906edb0fa976659e7eb7389f"
repo-rev = "main"
repo-url = "https://github.com/Circuitscape/Omniscape.jl.git"
uuid = "a38d70fc-99f5-11e9-1e3d-cbca093024c3"
version = "0.5.3"
[[OpenJpeg_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "LittleCMS_jll", "Pkg", "libpng_jll"]
git-tree-sha1 = "76374b6e7f632c130e78100b166e5a48464256f8"
uuid = "643b3616-a352-519d-856d-80112ee9badc"
version = "2.4.0+0"
[[OrderedCollections]]
git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.1"
[[PROJ_jll]]
deps = ["Artifacts", "JLLWrappers", "LibCURL_jll", "LibSSH2_jll", "Libdl", "Libtiff_jll", "MbedTLS_jll", "Pkg", "SQLite_jll", "Zlib_jll", "nghttp2_jll"]
git-tree-sha1 = "2435e91710d7f97f53ef7a4872bf1f948dc8e5f8"
uuid = "58948b4f-47e0-5654-a9ad-f609743f8632"
version = "700.202.100+0"
[[Pardiso]]
deps = ["Libdl", "LinearAlgebra", "MKL_jll", "SparseArrays"]
git-tree-sha1 = "a30f2daa4eb68ff2c1eb55419d58135c2d46de86"
uuid = "46dd5b70-b6fb-5a00-ae2d-e8fea33afaf2"
version = "0.5.3"
[[Parsers]]
deps = ["Dates"]
git-tree-sha1 = "438d35d2d95ae2c5e8780b330592b6de8494e779"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.0.3"
[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.2"
[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
[[ProgressMeter]]
deps = ["Distributed", "Printf"]
git-tree-sha1 = "afadeba63d90ff223a6a48d2009434ecee2ec9e8"
uuid = "92933f4c-e287-5a05-a399-4b506db050ca"
version = "1.7.1"
[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
[[RecipesBase]]
git-tree-sha1 = "44a75aa7a527910ee3d1751d1f0e4148698add9e"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.1.2"
[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
[[SQLite_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "9a0e24b81e3ce02c4b2eb855476467c7b93b8a8f"
uuid = "76ed43ae-9a5d-5a62-8c75-30186b810ce8"
version = "3.36.0+0"
[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"
[[SimpleTraits]]
deps = ["InteractiveUtils", "MacroTools"]
git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231"
uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d"
version = "0.9.4"
[[SimpleWeightedGraphs]]
deps = ["LightGraphs", "LinearAlgebra", "Markdown", "SparseArrays", "Test"]
git-tree-sha1 = "f3f7396c2d5e9d4752357894889a87340262f904"
uuid = "47aef6b3-ad0c-573a-a1e2-d07658019622"
version = "1.1.1"
[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"
[[SodiumSeal]]
deps = ["Base64", "Libdl", "libsodium_jll"]
git-tree-sha1 = "80cef67d2953e33935b41c6ab0a178b9987b1c99"
uuid = "2133526b-2bfb-4018-ac12-889fb3908a75"
version = "0.1.1"
[[SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.0.1"
[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
[[StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "3240808c6d463ac46f1c1cd7638375cd22abbccb"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.2.12"
[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
[[StatsAPI]]
git-tree-sha1 = "1958272568dc176a1d881acb797beb909c785510"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.0.0"
[[StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "fed1ec1e65749c4d96fc20dd13bea72b55457e62"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.9"
[[SuiteSparse]]
deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"]
uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9"
[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
[[TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.1"
[[Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"]
git-tree-sha1 = "d0c690d37c73aeb5ca063056283fde5585a41710"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.5.0"
[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
[[TimeZones]]
deps = ["Dates", "Future", "LazyArtifacts", "Mocking", "Pkg", "Printf", "RecipesBase", "Serialization", "Unicode"]
git-tree-sha1 = "6c9040665b2da00d30143261aea22c7427aada1c"
uuid = "f269a46b-ccf7-5d73-abea-4c690281aa53"
version = "1.5.7"
[[URIs]]
git-tree-sha1 = "97bbe755a53fe859669cd907f2d96aee8d2c1355"
uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
version = "1.3.0"
[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"
[[Zstd_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "cc4bf3fdde8b7e3e9fa0351bdeedba1cf3b7f6e6"
uuid = "3161d3a3-bdf6-5164-811a-617609db77b4"
version = "1.5.0+0"
[[libgeotiff_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libtiff_jll", "PROJ_jll", "Pkg"]
git-tree-sha1 = "a5cc2e3dd7b1c1e783a61b8ab7de03eebddfed60"
uuid = "06c338fa-64ff-565b-ac2f-249532af990e"
version = "1.6.0+1"
[[libpng_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "94d180a6d2b5e55e447e2d27a29ed04fe79eb30c"
uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f"
version = "1.6.38+0"
[[libsodium_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "848ab3d00fe39d6fbc2a8641048f8f272af1c51e"
uuid = "a9144af2-ca23-56d9-984f-0d03f7b5ccf8"
version = "1.0.20+0"
[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"
[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"Hi, sorry, I'm not the most proficient with programming. How does one use these files to install the correct versions in Julia?
ptfreeman-csp
commented
1 year ago @coport I followed the section of this tutorial to recreate the environment that @gagecarto was working with.
mir123
commented
1 year ago Confirmed that using @gagecarto 's environment wih Julia 1.6.2 the job runs successfuly. As posted on the issue in Circuitscape, the latest version of Circuitscape does not fix the issue.
slamander
commented
1 year ago Have there been any developments on this issue? I'm running my job on a university supercomputer, and therefore changing the environment to match @gagecarto 's may be quite convoluted. Is there any way to access old versions without manually doing so?
FanchFanchec
commented
1 year ago Hello Same issue for me I reproduce @gagecarto environment with Julia 1.6.2 (section of [this tutorial]) but..." nested task error: AssertionError: norm(G * v .- curr, Inf) / norm(v, Inf) < 1.0e-5" So many tries.
max-mapper
commented
1 year ago We have had some luck getting rid of this error by removing "islands" of permeable pixels (>=1) that were surrounded by a "moat" of impervious pixels (NoData/source strength 0). In some cases the radius of the moving window seemed to get stuck in these islands and was causing this error. We changed our rasters to avoid creating islands and haven't had the error since. Sorry I don't have a good reproducible test case handy
ptfreeman-csp
commented
1 year ago That certainly could square with the issues we were having given that we had many permeable pixel islands surrounded by moats of impervious pixels!
On Thu, Aug 10, 2023 at 3:28 PM Max @.***> wrote:
We have had some luck getting rid of this error by removing "islands" of permeable pixels (>1) that were surrounded by a "moat" of impervious pixels (NoData/source strength 0). In some cases the radius of the moving window seemed to get stuck in these islands and was causing this error. We changed our rasters to avoid creating islands and haven't had the error since. Sorry I don't have a good reproducible test case handy
— Reply to this email directly, view it on GitHub https://github.com/Circuitscape/Omniscape.jl/issues/127#issuecomment-1674006000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUE56Z7IDUNP4KK6BJMQYRTXUVOAJANCNFSM6AAAAAASLSAFGM . You are receiving this because you were mentioned.Message ID: @.***>
slamander
commented
1 year ago Does anyone have advice on locating and mitigating these permeable pixel islands? With quite large rasters, I can't comb through values efficiently or effectively.
vlandau
commented
1 year ago Ultimately this issue lies with Circuitscape, so I'm going to close this given that this issue has been discussed over there. If someone can create a minimum working example of the "pixel island" issue in a single Advanced mode Circuitscape run, that may confirm whether this is the root cause, and will likely help speed up any potential resolution to the issue. The reason that this wasn't showing up in older versions of Omniscape is because those were using older versions of Circuitscape, which didn't include this assert check (which basically us used to confirm that the solution arrived at by the linear solver is close to correct). Once Omniscape's dep on Circuitscape was bumped up to the version that included those checks, this issue came to the surface.
BortEdwards
commented
8 months ago Have just run into this same issue (julia/1.9.4) for runs with 25 or 50 mile windows using resistance surfaces that are fine for smaller windows. Tinkered with small changes in block size, hoping it might nudge the windows past some problematic pixel arrangement, but to no effect. Am not relishing the idea of having to mess around with a painfully arrived at series of window/block cell sizes. Hopefully I can identify the cells and try @slamanders method of fixing them individually... Appreciate everyone's work, but frustrating there has been no fix :/
vlandau
commented
8 months ago @BortEdwards There has been an attempted fix in Circuitscape.jl, just released today. You'll see above your comment that one of the recent patches to Circuitscape mentioned this issue. Try updating Circuitscape to v5.13.3.
This seems similar but not quite like #108 and #100, also see https://github.com/Circuitscape/Circuitscape.jl/issues/305
Omniscape crashed at 91% of a run with a TaskFailedException. This is a fairly large job, with a 224M cells raster (resolution 30m) and a 2200 cell radius (block size 221). Memory usage with 32 cores topped at around 360 GB.
I had a successful run with the same resistance and source surfaces using a 200 cell radius (block size 21). Reducing resolution of surfaces to 90m with a radius of 777 also runs successfully.
Julia: 1.8.3 Omniscape: 0.5.8
config:
Keeping the default cg+amg solver.
Resistance file Source file
Error message: