search

Circuitscape / Omniscape.jl

Functions to compute omnidirectional landscape connectivity using circuit theory and the Omniscape algorithm.
https://docs.circuitscape.org/Omniscape.jl/stable/
MIT License
58 stars 12 forks source link

ERROR: TaskFailedException: Adjacency / distance matrices must be symmetric #102

Closed linhunt closed 1 year ago

linhunt commented 3 years ago

Good Morning and Happy Monday!

After about 14 days and at 97% A large run I was performing crashed giving me this error message. Like in my other runs I have just one input file which has the resistances and the sources and targets are derived from this one file. Also this is the same file I have successfully run omniscape on many times. This run just has different resistance values, and different block/radius settings. I have also successfully produced Omniscape outputs for my study area at the same block/radius settings as this run. 

██████████████████████ |  ETA: 10:00:5Progress:  97%|███████████████████████████████████████ |  ETA: 9:43:3 ERROR: TaskFailedException:
Adjacency / distance matrices must be symmetric
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] SimpleWeightedGraph at C:\Users\lhunt\.julia\packages\SimpleWeightedGraphs\IDzOp\src\simpleweightedgraph.jl:16 [inlined]
 [3] SimpleWeightedGraph at C:\Users\lhunt\.julia\packages\SimpleWeightedGraphs\IDzOp\src\simpleweightedgraph.jl:27 [inlined]
 [4] compute_advanced_data(::Circuitscape.RasData{Float64,Int64}, ::Circuitscape.RasterFlags, ::Dict{String,String}) at C:\Users\lhunt\.julia\packages\Circuitscape\9x9VD\src\raster\advanced.jl:56
 [5] calculate_current(::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Array{Float64,2}, ::Circuitscape.RasterFlags, ::Dict{String,String}, ::DataType) at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\utils.jl:360
 [6] solve_target!(::Int64, ::Int64, ::Dict{String,Int64}, ::Array{Float64,2}, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Omniscape.OmniscapeFlags, ::Dict{String,String}, ::Circuitscape.RasterFlags, ::Circuitscape.OutputFlags, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::String, ::String, ::Float64, ::Float64, ::Float64, ::Float64, ::Array{Float64,2}, ::Array{Float64,3}, ::Array{Float64,3}, ::DataType) at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\utils.jl:507
 [7] macro expansion at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\main.jl:273 [inlined]
 [8] (::Omniscape.var"#71#threadsfor_fun#15"{Dict{String,Int64},DataType,Omniscape.OmniscapeFlags,String,String,Float64,Float64,Float64,Float64,Dict{String,String},Int64,Circuitscape.OutputFlags,Circuitscape.RasterFlags,ProgressMeter.Progress,Int64,UnitRange{Int64}})(::Bool) at .\threadingconstructs.jl:81
 [9] (::Omniscape.var"#71#threadsfor_fun#15"{Dict{String,Int64},DataType,Omniscape.OmniscapeFlags,String,String,Float64,Float64,Float64,Float64,Dict{String,String},Int64,Circuitscape.OutputFlags,Circuitscape.RasterFlags,ProgressMeter.Progress,Int64,UnitRange{Int64}})() at .\threadingconstructs.jl:48
Stacktrace:
 [1] wait at .\task.jl:267 [inlined]
 [2] threading_run(::Function) at .\threadingconstructs.jl:34
 [3] macro expansion at .\threadingconstructs.jl:93 [inlined]
 [4] run_omniscape(::Dict{String,String}, ::Array{Union{Missing, Float64},2}; reclass_table::Array{Union{Missing, Float64},2}, source_strength::Array{Union{Missing, Float64},2}, condition1::Array{Union{Missing, Float64},2}, condition2::Array{Union{Missing, Float64},2}, condition1_future::Array{Union{Missing, Float64},2}, condition2_future::Array{Union{Missing, Float64},2}, wkt::String, geotransform::Array{Float64,1}, write_outputs::Bool) at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\main.jl:268
 [5] top-level scope at REPL[7]:1
vlandau commented 3 years ago

Ugh, it is never nice to get errors so late in a large compute task. Sorry about this. It appears that something is happening in Circuitscape's construction of the graphs, which could either be due to a bug in Circuitscape, or an issue with an input that Omnsicape is passing to Circuitscape.

@ranjanan is there any reason that Circuitscape would produce a non-symmetric adjacency matrix? This certainly seems like a strange edge case. I guess that's one of the challenges with Omniscape -- thousands of CIrcuitscape solves for a given problem will no doubt reveal edge cases! Large problems also make these things hard to reproduce. Maybe when an error occurs, I should catch it in Omniscape and print the row/column indices for the moving window that is being solved so it can be reproduced. This should be easy enough to do -- in fact I'm going to do this now. It will also help with debugging errors related to matrix norm checks failing.

ranjanan commented 3 years ago

Yeah sorry @linhunt, this is never nice. Vincent, if you could track down the particular rows, if you can track down the portion of the moving window which is causing this I can then debug on the Circuitscape. It shouldn't produce an asymmetric matrix in any case.

linhunt commented 3 years ago

Hey, no worries! I appreciate you working on debugging this.

Lindsay

From: Ranjan Anantharaman @.> Sent: Monday, August 2, 2021 11:56 AM To: Circuitscape/Omniscape.jl @.> Cc: Hunt, Lindsay E @.>; Mention @.> Subject: [EXTERNAL] Re: [Circuitscape/Omniscape.jl] ERROR: TaskFailedException: Adjacency / distance matrices must be symmetric (#102)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Yeah sorry @linhunthttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flinhunt&data=04%7C01%7Clhunt%40usgs.gov%7Cd176a93778054141c38208d955d691fc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637635202323302017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tSt7XT6O1ov9iJT4T3XZYFai7DmrZb4C499j3tqJMUg%3D&reserved=0, this is never nice. Vincent, if you could track down the particular rows, if you can track down the portion of the moving window which is causing this I can then debug on the Circuitscape. It shouldn't produce an asymmetric matrix in any case.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FCircuitscape%2FOmniscape.jl%2Fissues%2F102%23issuecomment-891179711&data=04%7C01%7Clhunt%40usgs.gov%7Cd176a93778054141c38208d955d691fc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637635202323302017%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=g2ZraBhUk46xWbJGS8KsB%2FW0rvYqD0QLrYuY45ulQLg%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJWIV5G3LQDYYMLJBS2GETTT23E43ANCNFSM5BMXVEKA&data=04%7C01%7Clhunt%40usgs.gov%7Cd176a93778054141c38208d955d691fc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637635202323311970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oq%2Fa7yfmaYRq2HQTu%2BLwfLUBlakEQHpL8%2FTw%2FzhkOmI%3D&reserved=0.

ranjanan commented 3 years ago

@vlandau we will have to run this job again to reproduce the error right? In that case, @linhunt would have to send us their files?

vlandau commented 3 years ago

Yes we would need to re-run, though it is unfortunate that it is for such a large problem. I should go ahead and issue a patch release for Omniscape now so the error catching is in the latest version.

linhunt commented 3 years ago

Hi,

Would it be easier/faster for you guys if I download the new version of Omniscape and try running the same problem again and see if I get the error again? Would that make it easier for you to find the error rather than me sending you a large file and you guys spending many days waiting for it to crash at the end?

Lindsay

From: Vincent Landau @.> Sent: Friday, August 6, 2021 11:02 AM To: Circuitscape/Omniscape.jl @.> Cc: Hunt, Lindsay E @.>; Mention @.> Subject: [EXTERNAL] Re: [Circuitscape/Omniscape.jl] ERROR: TaskFailedException: Adjacency / distance matrices must be symmetric (#102)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Yes we would need to re-run, though it is unfortunate that it is for such a large problem. I should go ahead and issue a patch release for Omniscape now so the error catching is in the latest version.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FCircuitscape%2FOmniscape.jl%2Fissues%2F102%23issuecomment-894358561&data=04%7C01%7Clhunt%40usgs.gov%7Cbb03738fdbad4b10735d08d958f3c38d%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637638628000703467%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=KxmM9PSO1ywV6%2B%2F5f0mM7uN7I2FfkvaPzrFLtFwoCF4%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJWIV5AA5OSYKIJ66SKHIFLT3QBQPANCNFSM5BMXVEKA&data=04%7C01%7Clhunt%40usgs.gov%7Cbb03738fdbad4b10735d08d958f3c38d%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637638628000713424%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=3sJ4jM%2BA80R03N9rk%2BDr1BIBulpJhTsHDq6v94HZivo%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Clhunt%40usgs.gov%7Cbb03738fdbad4b10735d08d958f3c38d%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637638628000713424%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Nym3D%2BEspomigr2KqNoMg1DYkp80U6St3zTUy4bXiUA%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7Clhunt%40usgs.gov%7Cbb03738fdbad4b10735d08d958f3c38d%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637638628000713424%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BL5wq6f%2B6mRBfzsYgaMga71MXXa%2BwhZdYE85UQytepM%3D&reserved=0.

vlandau commented 3 years ago

That would be very helpful, but only if it is not costing you extra money to run the problem again. Also, how many threads are you running it on? we might be able to get it done faster since @ranjanan might have access to some more powerful computers, but I'm not sure.

Basically the new error handling will print out the index of the target pixel that is failing (the row and column for that pixel) so then we can reproduce the problem on a much smaller scale by clipping the landscape just to what we need. To do that, we'll still eventually need the full resistance surface and other raster inputs.

vlandau commented 1 year ago

@linhunt Is this still giving you problems or have you been able to reproduce it and get a column/row index?

vlandau commented 1 year ago

Closing due to lack of updates. Please reopen if it becomes an issue again!