Circuitscape in Julia unable to process a 1.4 Billion pixel landscape

[IndranilM0ndal]

Hello, I am new to Circuitscape in Julia and also Julia per se. I am trying to run Circuitscape on a landscape which is 37946 X 36946 pixels of 30m spatial resolution. I have scaled the conductance values from 1 to 100.

The specifications of my system is: 2 X Intel Xeon Gold 6136 so total 24 physical processor cores 128GB RAM.

When I tried parallel processing using 10 cores, my system ran out of memory. Then I disabled parallel processing and ran it. First it took a lot of time to read the maps even on the system that I am using, which I thought has a good specification. Then it gave this new error:

ArgumentError: the index type Int32 cannot hold 4075170303 elements; use a larger index type

I am attaching a screen shot of the console.

Regards, Indranil.

[ViralBShah]

1.4 billion just sounds really big. You should try with 1 core first. And use the 64-bit indexing.

[IndranilM0ndal]

Hi Viral. Thanks for the reply. I have upgraded my RAM to 348GB now. Still I will go with 1 core first as you suggested. Issue #200 provided some help. Will be using the following to use 64 bit indexing: use_64bit_indexing = true

[ViralBShah]

Great. Parallel processing also needs extra memory. So the larger the problem, the more you have to start slowly and push the parallelism.

[IndranilM0ndal]

Hi Viral. Ran the same landcape with 1 core and 64 bit indexing. FAILED !! OutOfMemoryError()

My conductance surface is 10.5GB and the core area raster is 7.69GB. I am unable to understand why 348GB RAM is not enough.

Attaching screenshot. Also, this is being run on Windows 10 Pro (if that information is of any use). I have access to a Linux machine as well. I can swap the RAM modules and try on that if that helps.

[ViralBShah]

Which solver are you using? Make sure it is cg+amg.

[ViralBShah]

The underlying graph and solver data structures easily take 10x-20x more memory than your surface.

We have done reasonably careful memory profiling, but there may be more options.

[IndranilM0ndal]

Which solver are you using? Make sure it is cg+amg.

I was using the cholmod solver. I will give it a try with cg+amg.

[ViralBShah]

Oh yeah - cholmod will not work for anything more than 1-10M cell landscapes. Maybe we should print a warning.

[IndranilM0ndal]

Was amazed to see the memory gradually maxing out and the application stopping with a OutOfMemoryError(). I have resampled my surfaces from 30m to 100m and saw a dramatic reduction in size. Now both the habitat and node rasters are less than 1GB. Trying again with resampled image.

[IndranilM0ndal]

Hi Viral, with reduced file sizes things seem to be running, even with 4 cores in parallel !! Processing time in each step has drastically reduced too along with a much lower memory footprint.

Of all the ini files provided with the test run, can you please point me to one which has all the settings that is honored by Circuitscape in Julia? In case there isn't one, can we compile an exhaustive list of all the setting in an ini file and keep it on the main GitHub page of Circuitscape.jl ? I also believe that it will be helpful if the ini file is annoted. Users who have graduated from the Python, ArcGIS or the Windows standalone version will have clues to what each settings mean, but it will be helpful for first time Circuitscape users who are taking their first go on Julia.

[ViralBShah]

The old files are fully compatible. Yes, we should document these things. Beyond what I shared here, there really are not many more new options in the ini files. Would you be able to submit a PR to the documentation or to this repo?

[IndranilM0ndal]

I am new to GitHub as well, so I will need to read up about pull requests before I am able to do so. However, I can easily prepare a template ini file with annotation.

[vlandau]

Close and move to #233?

[IndranilM0ndal]

Once I resampled and made my landscape smaller in MBs, Circuitscape in Julia started working without any error. In fact the analysis that I started is still running. It has 19,900 pairs, and at this rate, I calculated, it will take 200 days!! I am running in 4 cores in parallel and about 50% of 384GB memory usage.

My question is how does this parallelism work? I was thinking that each core will be processing a pair at a time, so 4 pairs will be processed at the same time. But I don't see that from the logs. I wanted to use parallelism to save me some time.

This is when the job started:

And this is right now:

[ViralBShah]

We need to move to the multi-threading version of circuitscape that will allow more parallelism and lesser memory. But it is not even ready to try yet, since we are running into some issues.

For now, you can make your problem even smaller and increase parallelism, and/or reduce the number of pairs. What mode are you using? Simple pairwise resistance, no polygons? The simpler you can make it the faster it will go.

[ViralBShah]

Close and move to #233?

We can keep this open to discuss the particular problem at hand.

[vlandau]

Are you using an include pairs file to specify which pairs you want connected? That can affect the scheduler and make parallelism less efficient (#165)

[IndranilM0ndal]

I am using pairwise mode and I am not using an include pairs file. I have made a raster from the polygon shapefile of the core area polygons and mentioned the path of the file in the "point_file" option. The contents of the ini file are as below. Please tell me if anything is extra or if I am missing anything.

[Options for advanced mode] ground_file_is_resistances = True source_file = None remove_src_or_gnd = keepall ground_file = None use_unit_currents = False use_direct_grounds = False

[Calculation options] low_memory_mode = False solver = cg+amg print_timings = True preemptive_memory_release = True

[Options for pairwise and one-to-all and all-to-one modes] included_pairs_file = None use_included_pairs = False point_file = D:\folder\raster_of_core_polygons.asc

[Output options] write_cum_cur_map_only = True log_transform_maps = False output_file = D:\folder\habitat_raster.out write_max_cur_maps = False write_volt_maps = False set_null_currents_to_nodata = False set_null_voltages_to_nodata = False compress_grids = False write_cur_maps = True set_focal_node_currents_to_zero = True

[Short circuit regions (aka polygons)] use_polygons = False polygon_file = None

[Connection scheme for raster habitat data] connect_four_neighbors_only = False connect_using_avg_resistances = False

[Habitat raster or graph] habitat_file = D:\cirascii\resistance.asc habitat_map_is_resistances = True

[Options for one-to-all and all-to-one modes] use_variable_source_strengths = False variable_source_file = None

[Version] version = unknown

[Mask file] use_mask = False mask_file = None

[Circuitscape mode] data_type = raster scenario = pairwise parallelize = True max_parallel = 8 use_64bit_indexing = True log_resource_usage_info = True

[vlandau]

Nothing jumps out to me as problematic in your .ini

There seems to be something going on with the task scheduler. @ranjanan or @ViralBShah would there be any reason that the scheduler would behave differently in Julia 1.4?

I'm seeing similar output patterns (that imply no parallel processing) for a specific test on Julia 1.4 as well.

[ Info: Testing sgVerify5
[ Info: 2020-05-05 16:57:56 : Precision used: Double
[ Info: 2020-05-05 16:57:56 : Starting up Circuitscape to use 2 processes in parallel
[ Info: 2020-05-05 16:57:59 : Reading maps
[ Info: 2020-05-05 16:57:59 : Resistance/Conductance map has 14 nodes
[ Info: 2020-05-05 16:57:59 : Total number of pair solves = 10
[ Info: 2020-05-05 16:57:59 : Solving pair 1 of 10
[ Info: 2020-05-05 16:57:59 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:57:59 : Graph has 10 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:57:59 : Time taken to construct preconditioner = 9.7036e-5 seconds
[ Info: 2020-05-05 16:57:59 : Time taken to construct local nodemap = 2.194e-6 seconds
[ Info: 2020-05-05 16:58:04 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:04 : Time taken to solve linear system = 0.003566931 seconds
[ Info: 2020-05-05 16:58:04 : Time taken to write voltage maps = 0.019530949 seconds
[ Info: 2020-05-05 16:58:04 : Time taken to write current maps = 0.319035418 seconds
[ Info: 2020-05-05 16:58:04 : Solving pair 2 of 10
[ Info: 2020-05-05 16:58:04 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:04 : Graph has 11 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:04 : Time taken to construct preconditioner = 4.5504e-5 seconds
[ Info: 2020-05-05 16:58:04 : Time taken to construct local nodemap = 2.733e-6 seconds
[ Info: 2020-05-05 16:58:04 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:04 : Time taken to solve linear system = 0.003556326 seconds
[ Info: 2020-05-05 16:58:04 : Time taken to write voltage maps = 0.071955798 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 0.289728361 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 3 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 11 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 5.6052e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.678e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.3273e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 9.7962e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 8.8905e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 4 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 11 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 4.4383e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.652e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.3623e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 6.7633e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 8.0678e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 5 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 11 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 3.3565e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.668e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.3035e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 7.5914e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 8.9053e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 6 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 11 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 2.8466e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.37e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.1571e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 5.1227e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 7.4459e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 7 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 11 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 4.1619e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.437e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 9.907e-6 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 5.9589e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 6.1835e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 8 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 12 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 4.0396e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.651e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.0052e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 5.0666e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 7.7561e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 9 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 12 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 4.1554e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.481e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.0198e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 6.8243e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 6.2656e-5 seconds
[ Info: 2020-05-05 16:58:05 : Solving pair 10 of 10
[ Info: 2020-05-05 16:58:05 : Solver used: AMG accelerated by CG
[ Info: 2020-05-05 16:58:05 : Graph has 12 nodes, 2 focal points and 1 connected components
[ Info: 2020-05-05 16:58:05 : Time taken to construct preconditioner = 2.7234e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to construct local nodemap = 2.451e-6 seconds
[ Info: 2020-05-05 16:58:05 : Scheduling pair 1 of 1 to be solved
[ Info: 2020-05-05 16:58:05 : Time taken to solve linear system = 1.0929e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write voltage maps = 7.0516e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to write current maps = 7.8654e-5 seconds
[ Info: 2020-05-05 16:58:05 : Time taken to complete job = 5.912580516
[ Info: Testing sgVerify5_cum_curmap.asc
[ Info: Test sgVerify5_cum_curmap.asc passed

[ranjanan]

@indranil-wii it seems that your point file contains polygons. That is, you have focal regions you'd like to collapse into one point in your point file. We do not support parallelism (yet) in this mode because you have to keep recomputing the graph datastructure for every solve and this would take a lot more memory if done in parallel (imagine holding 4 of these large graph datastructures in memory).

Is there a way you can make your point files have points instead of regions? (Use the centroids of those focal regions, for example. @vlandau you may have better advice on this). If you do this, the parallelism will work and your problem may be solved faster.

[vlandau]

Use the centroids of those focal regions, for example

I would do exactly that.

[IndranilM0ndal]

I could do that to simplify the problem and conceptually too it seems fine to use the centroids, However, there are practical problems. The corridors won't be mapped right. On ground the movement of animals is not directed towards the centroid of a park from outside but towards the nearest place on the perimeter. So I have to preserve the actual shape of the core area polygon.

[vlandau]

You could set the sources to be centroids and set your original source polygons as short circuit regions. I believe that would resolve the issue and current (i.e. movement) would be directed toward the nearest polygon edge instead of the centroid. @ranjanan would that work with parallelism?

[ViralBShah]

@indranil-wii Can we request you to take the learnings from this discussion and add them to the README for now https://github.com/Circuitscape/Circuitscape.jl/blob/master/README.md?

There is a little pen next to the README file, so you can just edit the file and submit some text. We can have a new section called - Notes for using Circuitscape on very large grids.

[IndranilM0ndal]

@indranil-wii Can we request you to take the learnings from this discussion and add them to the README for now https://github.com/Circuitscape/Circuitscape.jl/blob/master/README.md?

There is a little pen next to the README file, so you can just edit the file and submit some text. We can have a new section called - Notes for using Circuitscape on very large grids.

Sure, I can do that.

[IndranilM0ndal]

@indranil-wii it seems that your point file contains polygons. That is, you have focal regions you'd like to collapse into one point in your point file. We do not support parallelism (yet) in this mode because you have to keep recomputing the graph datastructure for every solve and this would take a lot more memory if done in parallel (imagine holding 4 of these large graph datastructures in memory).

Is there a way you can make your point files have points instead of regions? (Use the centroids of those focal regions, for example. @vlandau you may have better advice on this). If you do this, the parallelism will work and your problem may be solved faster.

I would like to try once with point files as my core area. However, I have never done that before. How do I format my core area file then, if they are to be points?

[ViralBShah]

The suggestion is not use the core area / polygons at all, and just compute resistance between pairs of central points in all your core areas.

[IndranilM0ndal]

The suggestion is not use the core area / polygons at all, and just compute resistance between pairs of central points in all your core areas.

Yes, I understand. In what format do I supply the central points file?

[vlandau]

Yes, I understand. In what format do I supply the central points file?

This is in the documentation. See the section titled "Focal node location and data type". Focal points can be in the same ASCII format. They just must be only one pixel each.

[ranjanan]

@ranjanan would that work with parallelism?

Should work.

[Kim-Hall]

Just read through this and while I can't help on the technical issues, I'm curious to know what is being modeled here, given there is such a huge number of pairs to connect. If you are connecting "everything to everything" maybe Omniscape would be worth a try? While Vincent is the expert, I'm happy to do a zoom overview of that tool and show you some examples to see if it might fit your problem. I'm assuming similar issues would arise given you still have a gigantic grid, but for that tool you would only be connecting "everything to everything" in a moving window, not trying to directly connect pairs that are long distances apart.

[vlandau]

This issue appears to have been sufficiently addressed and gone somewhat stale. I'm going to close for now. Can reopen later if needed.