search

eqcorrscan / EQcorrscan

Earthquake detection and analysis in Python.
https://eqcorrscan.readthedocs.io/en/latest/
Other
163 stars 86 forks source link

Parallel processing in write_correlations is very memory expensive #361

Closed emilyws1 closed 4 years ago

emilyws1 commented 4 years ago

Describe the bug utils.catalog_to_dd.write_correlations creates a list of sub-catalogs in memory for each "master" event in a catalog - this ends up being really memory intensive and for a clustered catalog of ~1600 events uses >50GB of RAM.

The offending line is here.

To Reproduce To follow.

Expected behavior Not eating all of the memory please. At-least serial processing should not create the list of sub-catalogs.

Desktop (please complete the following information):

Additional context This bug report was written by @calum-chamberlain

calum-chamberlain commented 4 years ago

This should be fixed by #374