search

HERA-Team / hera_qm

HERA Data Quality Metrics
MIT License
2 stars 2 forks source link

Break file loading in ant_metrics into its own function to prevent memory leaks #404

Closed jsdillon closed 2 years ago

jsdillon commented 2 years ago

Right now, ant_metrics is taking more than 128 GB of memory when running on 10 files and appears to be leaking memory quite badly:

image

This PR breaks file loading into a function, which is looped over per files group, to help make sure everything goes out of scope as appropriate.

jsdillon commented 2 years ago

And here's the memory profile on the same files with this branch:

image

codecov[bot] commented 2 years ago

Codecov Report

Merging #404 (54a1f6f) into master (1d2d156) will increase coverage by 0.00%. The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #404   +/-   ##
=======================================
  Coverage   97.05%   97.06%           
=======================================
  Files          10       10           
  Lines        3299     3306    +7     
=======================================
+ Hits         3202     3209    +7     
  Misses         97       97
Impacted Files Coverage Δ
hera_qm/ant_metrics.py 97.66% <100.00%> (+0.06%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1d2d156...54a1f6f. Read the comment docs.