HERA-Team / hera_qm

HERA Data Quality Metrics
MIT License
2 stars 2 forks source link

Improve ant_metrics handling of zeros #438

Closed jsdillon closed 1 year ago

jsdillon commented 1 year ago

A recent PR #434 simplified ant_metrics. calc_corr_stats but made it less tolerant of zeros appearing in either the odd or even spectrum (or both). As it stands, a single zero in either the even or odd spectrum makes the whole statistic NaN.

If a single X-engine is down, for example, it should still be possible to calculate a corr metric for a given baselines, even if there are some zeros. And in general, we want to do this to help identify white rabbit failures and cross-polarized antennas, which is the purpose of ant_metrics. This PR robustifies the calculation using a using a nanmean.

If there are zeros in the spectrum, ant_class. even_odd_zeros_checker is the right tool to diagnose that. If an antenna is totally dead, we already have a way to handle that in ant_metrics.

This PR adds a test to expose this issue. It also speeds up calc_corr_stats by ~15% by avoiding repeating the multiplication and normalization.

codecov[bot] commented 1 year ago

Codecov Report

Base: 97.10% // Head: 97.06% // Decreases project coverage by -0.04% :warning:

Coverage data is based on head (1f5b315) compared to base (87299d5). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #438 +/- ## ========================================== - Coverage 97.10% 97.06% -0.05% ========================================== Files 9 10 +1 Lines 3494 3510 +16 ========================================== + Hits 3393 3407 +14 - Misses 101 103 +2 ``` | [Impacted Files](https://codecov.io/gh/HERA-Team/hera_qm/pull/438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team) | Coverage Δ | | |---|---|---| | [hera\_qm/ant\_metrics.py](https://codecov.io/gh/HERA-Team/hera_qm/pull/438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9xbS9hbnRfbWV0cmljcy5weQ==) | `99.52% <100.00%> (-0.01%)` | :arrow_down: | | [hera\_qm/\_\_init\_\_.py](https://codecov.io/gh/HERA-Team/hera_qm/pull/438/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team#diff-aGVyYV9xbS9fX2luaXRfXy5weQ==) | `88.23% <0.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HERA-Team)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.