TillF
commented
7 years ago relates to #16
Closed lbergi closed 7 years ago
TillF
commented
7 years ago relates to #16
lbergi
commented
7 years ago I fixed the region of the dataset and converted any DCELL and FCELL data types to CELL. area2catena() is now succesfull, but prof_class() results to an error, that might also be a sign for RAM limitation.
This is the error message I got:
Till optimized the code, so that the file can be successfully loaded and processed. Still the data reaches a limit in R, the r session aborts without a message. I tried it with terminal only to gather some more information. This is the (shortened with [...] )output:
[....]
[1] "...100%"
...loaded.
profile clustering [...]
three closest catenas to cntre of class 1 [...]
three closest catenas to cntre of class 2 [...]
three closest catenas to cntre of class 3 [...]
start decomposition of TCs
partitioning class 1 of 3
Killed.Kind regards, Lisa
Dear Tobias and Till, We found out something about study area size and RAM limitation working with lumpR, that might be interesting for you: My study area is quite large: with 9090 m resolution it has 75545028 cells. (details further down). Area2catena works with 6 layers, that means that
Total Cells: 75545028 Layer (6)= 453 270 168 Cells are used, transfered and processed in the function. That process reaches the limits of my computer RAM (8 GB). Using the recommend parallelization (more cores) leads to some kind of an overflow of the RAM requirements. Observing a task manager you can see, that the CPU drops to 1-5% while RAM is full (95%). The function will never end (I interrupted it after 5 days of computing during the long easter weekend). You need to force break the whole machine. The solution that worked for us is to only use 1 core. With that, the function is successfully computed. But: the computation duration is about 12 h and the RAM is at limit (nothing works anymore, R session gives you the dubious "cannot allocate memory" message with any action you try and you need to restart the computer). A possible explanation is, that by needing more RAM than your computer offers, processes are transferred to swap (deferred for later). Transferring data to swap and back needs much time, what can be an explanation for long computing duration.
Summary: For study areas larger than ours you might use a server or a computer that has more RAM than 8 GB.
Details: Ehas= ca 28.000 (with the parameter settings quite all of them are included)
Region: projection: 1 (UTM) zone: -23 datum: wgs84 ellipsoid: wgs84 north: 8394608.94608946 south: 7573260.73260733 west: 150000 east: 895015.88956751 nsres: 90.00090001 ewres: 89.99950345 rows: 9126 cols: 8278 cells: 75545028
+----------------------------------------------------------------------------+
DATA:
Digital Elevation Model:
+----------------------------------------------------------------------------+
Flow Accum
+----------------------------------------------------------------------------+
Eha
+----------------------------------------------------------------------------+
_distriv
+----------------------------------------------------------------------------+
_elevriv
+----------------------------------------------------------------------------+
svc
+----------------------------------------------------------------------------+
Kind regards, Lisa and Josee