HERA-Team / hera_cal

Library for HERA data reduction, including redundant calibration, absolute calibration, and LST-binning.
MIT License
11 stars 8 forks source link

Improve memory usage in RFI_delay_slope_cal #850

Closed jsdillon closed 1 year ago

jsdillon commented 1 year ago

I noticed when running notebooks at NRAO that I was getting memory errors I didn't expect. Doing an mprof, I found a spike which I traced back to RFI_delay_slope_cal

image

This PR takes part of of the calculation of the per-baseline delay in RFI cal out of np.einsum and puts it into a loop to improve memory usage. It's about 15% slower (on my laptop, it's 2.85 seconds vs. 2.48 seconds), but a factor of Nrfi_chans more memory efficient, which seems like a great trade-off.

codecov[bot] commented 1 year ago

Codecov Report

Base: 97.18% // Head: 97.18% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (fd0651d) compared to base (9b7c3f4). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #850 +/- ## ======================================= Coverage 97.18% 97.18% ======================================= Files 19 19 Lines 8633 8635 +2 ======================================= + Hits 8390 8392 +2 Misses 243 243 ``` | [Impacted Files](https://codecov.io/gh/HERA-Team/hera_cal/pull/850?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team) | Coverage Δ | | |---|---|---| | [hera\_cal/abscal.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/850/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvYWJzY2FsLnB5) | `96.82% <100.00%> (+<0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

jsdillon commented 1 year ago

Here's the after:

image