Pysmurf op data quality checks - Githubissues

simonsobs / socs

Simons Observatory specific OCS agents.

BSD 2-Clause "Simplified" License

12 stars 12 forks source link

Pysmurf op data quality checks #675

Closed jlashner closed 3 weeks ago

jlashner commented 1 month ago

This PR adds quantiles for stats into session data for iv and bias step ops, and publishes them, so we can have some data quality metrics in grafana.

Description

Adds quantiles from Rn to IV op, and Rtes, Rfrac, and Si for bias steps / bias waves ops.

Motivation and Context

We would like to be able to tell immediately from grafana if there is an issue with detector data quality.

How Has This Been Tested?

Not yet tested. I'd like to test on satp1 or satp3 before merging.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[X] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

jlashner commented 1 month ago

This has now been tested on satp1, and quantiles are viewable in grafana:

We need to figure out a good way to display these and set alarms on them, but I think this is ready for review @BrianJKoopman.