search

Circuitscape / Omniscape.jl

Functions to compute omnidirectional landscape connectivity using circuit theory and the Omniscape algorithm.
https://docs.circuitscape.org/Omniscape.jl/stable/
MIT License
58 stars 12 forks source link

Errored pkg.test() and run_omniscape() returns "BoundsError" #62

Closed slamander closed 4 years ago

slamander commented 4 years ago

Hi CS and OS community,

I'm attempting to run OS (v 0.4.0) through R (v 4.0.1), however it always returns the same error (see Error 1). So, I then ran a Pkg.test in Julia, which returned a testing error (see Error 2). I then installed a dev version, which passed the package test, but when I attempt to run it on my own data (through R or Julia), it throws Error 1 again. I have almost 300 focal raster grids I would like to analyze, but I've uploaded a single example grid & .ini here.

BTW--because I have no idea what I'm doing, I created my .ini file to be proportional to block_size, radius, and buffer parameters values set in https://github.com/Circuitscape/Omniscape.jl/blob/main/test/input/config4.ini.

Error 1

Error: Error happens in Julia.
BoundsError: attempt to access 29935×29935 Array{Float64,2} at index [-31798:349, -31798:268]
Stacktrace:
 [1] throw_boundserror(::Array{Float64,2}, ::Tuple{UnitRange{Int64},UnitRange{Int64}}) at .\abstractarray.jl:541
 [2] checkbounds at .\abstractarray.jl:506 [inlined]
 [3] view at .\subarray.jl:158 [inlined]
 [4] maybeview at .\views.jl:133 [inlined]
 [5] dotview at .\broadcast.jl:1160 [inlined]
 [6] get_source(::Array{Float64,2}, ::Dict{String,Int64}; x::Int64, y::Int64, strength::Float64) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:99
 [7] calc_correction(::Dict{String,Int64}, ::Dict{String,String}, ::Circuitscape.OutputFlags) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:566
 [8] run_omniscape(::String) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\run_omniscape.jl:135
 [9] top-level scope at none:1
 [10] eval(::Module, ::Any) at .\boot.jl:331
 [11] eval_string(::String) at C:\Users\jbaec

Error 2

julia> Pkg.test("Omniscape")
    Testing Omniscape
Status `C:\Users\jbaecher\AppData\Local\Temp\jl_CJ8m5j\Project.toml`
  [2b7a1792] Circuitscape v5.7.0
  [a38d70fc] Omniscape v0.1.4
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [8dfed614] Test
Status `C:\Users\jbaecher\AppData\Local\Temp\jl_CJ8m5j\Manifest.toml`
  [2169fc97] AlgebraicMultigrid v0.3.0
  [c9ce4bd3] ArchGDAL v0.4.1
  [ec485272] ArnoldiMethod v0.0.4
  [fa961155] CEnum v0.4.1
  [2b7a1792] Circuitscape v5.7.0
  [34da2185] Compat v3.14.0
  [aa819f21] CompatHelper v1.13.6
  [9a962f9c] DataAPI v1.3.0
  [9a8bc11e] DataStreams v0.4.2
  [864edb3b] DataStructures v0.18.2
  [e2ba6199] ExprTools v0.1.1
  [8f5d6c58] EzXML v1.1.0
  [add2ef01] GDAL v1.1.4
  [a7073274] GDAL_jll v3.0.4+2
  [d604d12d] GEOS_jll v3.8.1+0
  [92fee26a] GZip v0.5.1
  [68eda718] GeoFormatTypes v0.3.0
  [cf35fbd7] GeoInterface v0.5.4
  [bc5e4493] GitHub v5.1.7
  [cd3eb016] HTTP v0.8.17
  [d25df0c9] Inflate v0.1.2
  [83e8ac13] IniFile v0.5.0
  [42fd0dbc] IterativeSolvers v0.8.4
  [682c06a0] JSON v0.21.0
  [aacddb02] JpegTurbo_jll v2.0.1+2
  [deac9b47] LibCURL_jll v7.71.1+0
  [29816b5a] LibSSH2_jll v1.9.0+3
  [94ce4f54] Libiconv_jll v1.16.0+6
  [89763e89] Libtiff_jll v4.1.0+1
  [093fc24a] LightGraphs v1.3.0
  [d3a379c0] LittleCMS_jll v2.9.0+0
  [1914dd2f] MacroTools v0.5.5
  [739be429] MbedTLS v1.0.2
  [c8ffd9c3] MbedTLS_jll v2.16.6+1
  [e1d29d7a] Missings v0.4.3
  [78c3b35d] Mocking v0.7.1
  [14a3606d] MozillaCACerts_jll v2020.7.22+0
  [a38d70fc] Omniscape v0.1.4
  [643b3616] OpenJpeg_jll v2.3.1+0
  [bac558e1] OrderedCollections v1.3.0
  [58948b4f] PROJ_jll v7.0.1+0
  [69de0a69] Parsers v1.0.10
  [3cdcf5f2] RecipesBase v1.0.2
  [76ed43ae] SQLite_jll v3.32.3+0
  [699a6c99] SimpleTraits v0.9.3
  [47aef6b3] SimpleWeightedGraphs v1.1.1
  [90137ffa] StaticArrays v0.12.4
  [f269a46b] TimeZones v1.3.2
  [ea10d353] WeakRefStrings v0.6.2
  [02c8fc9c] XML2_jll v2.9.10+2
  [83775a58] Zlib_jll v1.2.11+16
  [3161d3a3] Zstd_jll v1.4.5+1
  [b53b4c65] libpng_jll v1.6.37+5
  [8e850ede] nghttp2_jll v1.40.0+2
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
ERROR: LoadError: MethodError: no method matching Circuitscape.RasterMeta(::Int64, ::Int64, ::Float64, ::Float64, ::Float64, ::Float64, ::Int64)
Closest candidates are:
  Circuitscape.RasterMeta(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at C:\Users\jbaecher\.julia\packages\Circuitscape\XdJRQ\src\io.jl:20
  Circuitscape.RasterMeta(::Int64, ::Int64, ::Float64, ::Float64, ::Float64, ::Float64, ::Array{Float64,1}, ::String) at C:\Users\jbaecher\.julia\packages\Circuitscape\XdJRQ\src\io.jl:20
Stacktrace:
 [1] calculate_current(::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Circuitscape.RasterFlags, ::Dict{String,String}) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:196
 [2] calc_correction(::Dict{String,Int64}, ::Dict{String,String}, ::Circuitscape.OutputFlags) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:585
 [3] run_omniscape(::String) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\run_omniscape.jl:135
 [4] top-level scope at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\test\runtests.jl:3
 [5] include(::String) at .\client.jl:457
 [6] top-level scope at none:6
in expression starting at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\test\runtests.jl:3
ERROR: Package Omniscape errored during testing
Stacktrace:
 [1] pkgerror(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:52
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1566
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:328
 [4] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:315
 [5] #test#61 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [6] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [7] #test#60 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
 [8] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
 [9] test(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
 [10] test(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
 [11] top-level scope at REPL[5]:1

Thanks a ton,

Alex.

slamander commented 4 years ago

Update: I reinstalled Julia and now Omniscape passes Pkg.test, however run_omniscape still returns the same error (see error 1 above). Just to confirm, the files I used are here.

vlandau commented 4 years ago

Hi @slamander sorry for my late reply here. For some reason I missed the notification when you posted the original issue! Tests are passing for me as well in Julia 1.5.0 -- glad you got that working.

I will get a chance to look into this issue more thoroughly tomorrow, but right off the bat, I am seeing [a38d70fc] Omniscape v0.1.4 in the error output above. It looks like you are using Omniscape v0.1.4 (not v0.4.0) and Circuitscape v5.7.0, and those two versions are not compatible. Usually the reason for an old version of Omniscape being used is due to Julia not being at the latest version.

Can you share with me the following info?

  1. what version of Julia are you using (also verify that this is the version linked to R)
  2. The output of Pkg.status("Omniscape") run both from R and from the Julia terminal directly

Thanks!

slamander commented 4 years ago

Hey @vlandau, no worries, I'm sure my incessant issue posting is driving you mad. Now that I've updated Julia and Omniscape, I believe you solved it! Here's what you asked for:

Can you share with me the following info?

  1. what version of Julia are you using (also verify that this is the version linked to R)

Julia:

julia> VERSION
v"1.5.1"

R:

R> julia_command('VERSION')
v"1.5.1"
  1. The output of Pkg.status("Omniscape") run both from R and from the Julia terminal directly

Julia:

julia> Pkg.status("Omniscape")
Status `C:\Users\jbaecher\.julia\environments\v1.5\Project.toml`
  [a38d70fc] Omniscape v0.4.0

R:

R> julia_installed_package("Omniscape")
[1] "0.4.0"

I no longer get the errors mentioned above, instead a more common OutOfMemoryError() error. Curious, are there any resources you can point me toward that give 'best practice' recommendations for setting block_size, radius, buffer, and/or any other Omniscape parameters, given the job?

Again, I can't thank you enough for being such a great resource.

Best,

-Alex.

vlandau commented 4 years ago

No worries at all! The out of memory error means that your machine doesn't have the memory resources needed to run given the problem size. This isn't significantly affected by block_size, radius, or buffer, unless you have an very large radius (>>1000 pixels). In my experience out of memory errors have been due to very large resistance grids. How many pixels is your resistance grid?

One way to reduce memory consumption is to use fewer parallel processes. More parallel processes -> larger memory requirements. You could start by running in with parallelize = false in your .ini to see if that helps.

I'm going to close this issue for now since the memory issue is a hardware problem and not an Omniscape issue per se, but feel free to continue to comment. Chatting on gitter might make sense for this since it's more about general usage, but either works for me!

slamander commented 4 years ago

Great, thanks @vlandau. This is very helpful information, and I appreciate you turning me to the gitter forum! Omniscape is an awesome tool!

-Alex.