HERA-Team / hera_cal

Library for HERA data reduction, including redundant calibration, absolute calibration, and LST-binning.
MIT License
12 stars 8 forks source link

Faster slice func #854

Closed steven-murray closed 1 year ago

steven-murray commented 1 year ago

This caches the result of polstr2num for a particular HERAData object because I was finding that when reading a single file from H6C, when getting the slices for each baseline, it took about 12 seconds, or ~20% of the total read time. When LST binning, this can be an even higher percentage because you might only read one out of the two times -- decreasing actual read time but not decreasing the number of baselines that need to be indexed.

This PR makes the indexing time essentially negligible.

FWIW, on a single-shot read (the above was done reading 350 baselines at a time, similar to what would be done in a real pipeline), about 1/3 of the read time is taken in doing copies of the data into the datacontainers. I wonder if it would be useful to have a non-copying option.

jsdillon commented 1 year ago

This looks quite reasonable (though I'm surprised by the level of speed-up you're seeing), but it's failing tests...

codecov[bot] commented 1 year ago

Codecov Report

Base: 97.13% // Head: 97.18% // Increases project coverage by +0.04% :tada:

Coverage data is based on head (75cc6bf) compared to base (9790053). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #854 +/- ## ========================================== + Coverage 97.13% 97.18% +0.04% ========================================== Files 19 19 Lines 8632 8643 +11 ========================================== + Hits 8385 8400 +15 + Misses 247 243 -4 ``` | [Impacted Files](https://codecov.io/gh/HERA-Team/hera_cal/pull/854?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team) | Coverage Δ | | |---|---|---| | [hera\_cal/io.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvaW8ucHk=) | `98.54% <100.00%> (+<0.01%)` | :arrow_up: | | [hera\_cal/redcal.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvcmVkY2FsLnB5) | `99.19% <0.00%> (+<0.01%)` | :arrow_up: | | [hera\_cal/abscal.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvYWJzY2FsLnB5) | `96.83% <0.00%> (+<0.01%)` | :arrow_up: | | [hera\_cal/apply\_cal.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvYXBwbHlfY2FsLnB5) | `94.28% <0.00%> (+0.02%)` | :arrow_up: | | [hera\_cal/utils.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvdXRpbHMucHk=) | `96.24% <0.00%> (+0.16%)` | :arrow_up: | | [hera\_cal/reflections.py](https://codecov.io/gh/HERA-Team/hera_cal/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9jYWwvcmVmbGVjdGlvbnMucHk=) | `92.68% <0.00%> (+0.56%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.