Memory usage is too damn high

UCL / pyCascadia

Implementation of GEBCO cookbook remove-restore and other cleaning of topography/bathymetry. Uses `pyGMT`.

Mozilla Public License 2.0

9 stars 0 forks source link

Memory usage is too damn high #30

Closed JamieJQuinn closed 3 years ago

JamieJQuinn commented 3 years ago

Currently the memory usage of the remove-restore algorithm is very high (on the order of tens of GB) when outputting a grid of very fine resolution (e.g. 10m). This should be more of a priority now that we're polishing the remove-restore functionality.

JamieJQuinn commented 3 years ago

This is actually not as big a problem as I thought it was. Results of some basic experiments watching top while running remove-restore:

n update grids	region size (deg)	grid spacing (deg)	approx memory usage (GB)
2	0.5 x 0.2	1e-3	5
2	0.5 x 0.2	1e-4	7
2	1 x 1	1e-3	6
2	1 x 1	1e-4	8
2	5 x 5	1e-3	8
2	5 x 5	1e-4	Out of memory... (>20GB)

@Devaraj-G:

What do you think?
Grid spacing of around 1e-4 is where we want to be, right?
What's the expected region size?

JamieJQuinn commented 3 years ago

This no longer appears to be an issue.