CS tries to solve thousands of pairs

[merijnvdb]

My Pkg.status returns Circuitscape v5.10.0 I do seem to have an issue in Julia that I also had in Arcmap, but in Arcmap I could fix it by unchecking 'write current maps'. I tried setting 'write current maps' to true or false in the .ini but still I get this problem; despite only having 434 nodes, the application starting to solve 20.000 pairs (at 5 seconds per pair, this would take 30 hours):

See here for a zip file containing my .ini file, my resistance raster and node raster: circuit.zip Below is some output in Julia.

julia> compute("D:\Merijn\Desktop\circuit\currentmap.ini") [ Info: 2021-11-03 13:27:44 : Precision used: Double [ Info: 2021-11-03 13:27:50 : Reading maps [ Info: 2021-11-03 13:27:59 : Resistance/Conductance map has 3462333 nodes ┌ Warning: 2021-11-03 13:28:02 : yllcorner is not the same └ @ Circuitscape C:\Users\merij.julia\packages\Circuitscape\a9Quh\src\logging.jl:25 [ Info: 2021-11-03 13:28:06 : Solver used: AMG accelerated by CG [ Info: 2021-11-03 13:28:07 : Graph has 3462333 nodes, 434 focal points and 82 connected components [ Info: 2021-11-03 13:28:08 : Total number of pair solves = 20506 [ Info: 2021-11-03 13:28:15 : Time taken to construct preconditioner = 4.6566057 seconds [ Info: 2021-11-03 13:28:15 : Time taken to construct local nodemap = 0.712351 seconds [ Info: 2021-11-03 13:28:17 : Solving pair 1 of 20506

[vlandau]

Just copying my response about the thousands of pairs being solved from the PR:

@merijnvdb it is solving that many pairs because there are 20506 possible sets of two elements from a set of 434 (i.e. the combinatoric, 434 choose 2). Would you mind sharing your INI file here so we can take a look?

Is your problem that it is writing the current maps when you don't want it to? I previously misunderstood your issue as Circuitscape writing the files when you just wanted the cumulative map. In order to get at cumulative current flow, you need to solve each individual pair whether you write them or not. So this should be expected behavior, so long as the individual pair-wise maps are not being written to your filesystem.

[merijnvdb]

I want a cumulative map, i don't need the individual ones.

In arcmap if i unchecked 'write current maps' but checked 'write cumulative current map only' it would only need to solve about 204 pairs, but there would be no output map in the directory.

Does this mean i should run both the singular current maps and the cumulative to get the cumulative map, even though this would take over 24h?

On Wed, Nov 3, 2021, 2:26 PM Vincent Landau @.***> wrote:

Just copying my response about the thousands of pairs being solved from the PR:

@merijnvdb https://github.com/merijnvdb it is solving that many pairs because there are 20506 possible sets of two elements from a set of 434 (i.e. the combinatoric, 434 choose 2). Would you mind sharing your INI file here so we can take a look?

Is your problem that it is writing the current maps when you don't want it to? In order to get at cumulative current flow, you need to solve each individual pair whether you write them or not.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Circuitscape/Circuitscape.jl/issues/332#issuecomment-959806391, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHJWMNU3SXHIV72YFFKMATUKGEFPANCNFSM5HJSHJIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[ranjanan]

The cumulative map is calculated by calculating the individual ones and adding them up. There is no shortcut there as far as I am aware. Why do you think it should solve only 204 pairs?

[vlandau]

Circuitscape will always solve every pairwise combination (the intended behavior for pairwise mode). write_cur_maps only determines whether each pairwise map should be saved. The cumulative current map is the sum of every pairwise map, so they need to be solved individually. The fact that ArcMap was only trying to solve 204 pairs suggests to me that something was going wrong (unless you were using an include_pairs file?).

There are a few options for you to speed things up.

1. Use parallel processing to solve multiple pairwise problems at once. Add parallelize = true and set max_parallel in the INI file to some number less than the total number of cores you have on your computer (e.g. max_parallel = 3 if you have a 4 core machine).
2. Try all-to-one or one-to-all modes. This will result in 434 solves rather than 20k +

[merijnvdb]

That makes sense, reading the user guide I suspected something else, because I work with focal points rather than nodes;

under Calculation modes: If there are n focal nodes, there will be n(n - 1)/2 calculations unless you’re using focal points (only one cell per focal node) and not mapping currents or voltages. In the latter case, we can do it in n calculations (much faster).

[vlandau]

In your case you have to map the currents in order to get the cumulative map, unfortunately. The later only applies if you just want pairwise resistances and no current map (pairwise or cumulative).

[merijnvdb]

Okay, last question;

I'm trying to build an .ini file with 5 parrallel cores, and using the 'cholmod' process (because this might speed it up?), but the resulting .ini files mentions nothing of cores used and the calculations options only mention "solver = cg+amg". Should I just override this and add the parrallellize functions you mentioned , and maybe change to "solver = cholmod"?

Thanks for all your help

[vlandau]

I think your maps might be too big to support cholmod, but for parallizing, your INI would look something like this (see the last two lines):

[Circuitscape Mode]
data_type = raster
scenario = pairwise

[Version]
version = 5.0.0

[Habitat raster or graph]
habitat_file = resistance.tif
habitat_map_is_resistances = true

[Connection Scheme for raster habitat data]
connect_four_neighbors_only = false
connect_using_avg_resistances = false

[Short circuit regions (aka polygons)]
use_polygons = false
polygon_file = False

[Options for advanced mode]
ground_file_is_resistances = true
source_file = (Browse for a current source file)
remove_src_or_gnd = keepall
ground_file = (Browse for a ground point file)
use_unit_currents = false
use_direct_grounds = false

[Mask file]
use_mask = false
mask_file = None

[Options for one-to-all and all-to-one modes]
use_variable_source_strengths = false
variable_source_file = None

[Options for pairwise and one-to-all and all-to-one modes]
included_pairs_file = (Browse for a file with pairs to include or exclude)
use_included_pairs = false
point_file = nodes.tif

[Output options]
write_cum_cur_map_only = true
log_transform_maps = false
output_file = currentmap
write_max_cur_maps = false
write_volt_maps = false
set_null_currents_to_nodata = false
set_null_voltages_to_nodata = false
compress_grids = false
write_cur_maps = false

parallelize = true
max_parallel = 3

Just fix the file paths.

cholmod might not have been working if you didn't delete solver = cg+amg from the original ini file (see above). I don't think cholmod will work though. I tried it on my machine (which has 32GB of RAM), and it ran our of memory.

[merijnvdb]

You're right Vincent, cholmod did not work unfortunately,

I'm running it normal now, trying with 3/6 cores in my i7 laptop. Strangely enough the 'time taken to solve linear systems' seems to take longer when using parallel mode than when using not...

Op wo 3 nov. 2021 om 16:20 schreef Vincent Landau @.***

:

I think your maps might be too big to support cholmod, but for parallizing, your INI would look something like this (see the last two lines):

[Circuitscape Mode]data_type = rasterscenario = pairwise [Version]version = 5.0.0 [Habitat raster or graph]habitat_file = resistance.tifhabitat_map_is_resistances = true [Connection Scheme for raster habitat data]connect_four_neighbors_only = falseconnect_using_avg_resistances = false [Short circuit regions (aka polygons)]use_polygons = falsepolygon_file = False [Options for advanced mode]ground_file_is_resistances = truesource_file = (Browse for a current source file)remove_src_or_gnd = keepallground_file = (Browse for a ground point file)use_unit_currents = falseuse_direct_grounds = false [Mask file]use_mask = falsemask_file = None [Options for one-to-all and all-to-one modes]use_variable_source_strengths = falsevariable_source_file = None [Options for pairwise and one-to-all and all-to-one modes]included_pairs_file = (Browse for a file with pairs to include or exclude)use_included_pairs = falsepoint_file = nodes.tif [Output options]write_cum_cur_map_only = truelog_transform_maps = falseoutput_file = currentmapwrite_max_cur_maps = falsewrite_volt_maps = falseset_null_currents_to_nodata = falseset_null_voltages_to_nodata = falsecompress_grids = falsewrite_cur_maps = false parallelize = truemax_parallel = 3

Just fix the file paths.

cholmod might not have been working if you didn't delete solver = cg+amg from the original ini file (see above). I don't think cholmod will work though. I tried it on my machine (which has 32GB of RAM), and it ran our of memory.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Circuitscape/Circuitscape.jl/issues/332#issuecomment-959956456, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHJWMLABIYPD7FXDXJA3OTUKGRQDANCNFSM5HJSHJIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[vlandau]

Glad it is working! Some overhead is to be expected when using parallel processing. As long as it's not 3x longer, you're still saving time, fortunately. Since things are working, I'm going to close this issue, but please reach out again if you run into any more problems!

[merijnvdb]

Thank you for all your help! I don't want to clutter the page and open another issue, but about the processing; I now have results like this; Solving pair 218 of 20506 [ Info: 2021-11-03 16:55:10 : Time taken to solve linear system = 11.7691907 seconds [ Info: 2021-11-03 16:55:45 : Time taken to calculate current maps = 34.6949719 seconds Solving pair 219 of 20506 etc.

Then without parallel processing, it shows the same, except for that time taken to solve linear system is usually faster (under 10 seconds).

I don't really understand what you mean by overhead, is circuitscape solving more pairs in parallel that are not shown?

I'm unsure now whether or not to use parallel processing, I do have 6 cores and am only using 3 but I don't know whether the cores on a personal laptop are sufficient or not.

Thanks again!

Op wo 3 nov. 2021 om 16:40 schreef Vincent Landau @.***

:

Glad it is working! Some overhead is to be expected when using parallel processing. As long as it's not 3x longer, you're still saving time, fortunately. Since things are working, I'm going to close this issue, but please reach out again if you run into any more problems!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Circuitscape/Circuitscape.jl/issues/332#issuecomment-960000313, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASHJWMPHKJ6ADIIQREJPXQTUKGT4PANCNFSM5HJSHJIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[vlandau]

That is strange. When running the same problem I noticed no slow downs. It could be that you are allocating more cores than your computer has. If you are running any other programs on your computer, that might be slowing things down as well. Try running it with 2 cores to see if that helps. If you're running on an intel 4 core machine with hyperthreading, then you may have better luck sticking to the number of physical cores, which intel machines is half of the number of the total "cores" on intel chips.