simonsobs / sotodlib

Simons Observatory: Time-Ordered Data processing library.
MIT License
16 stars 19 forks source link

Add various preprocess plots #930

Closed davidvng closed 2 months ago

davidvng commented 3 months ago

Trending flags

- name: "trends"
      calc:
        max_trend: 2.5
        n_pieces: 10
      save: True
      plot: True
      select:
        kind: "any"

image

Jumps stats

- name: "jumps"
      calc:
        function: "twopi_jumps"
      save:
        jumps_name: "jumps_2pi"
      plot: True
      select:
        max_n_jumps: 5

image

Glitch stats

- name: "glitches"
      calc:
        t_glitch: 0.00001
        buffer: 10
        hp_fc: 1
        n_sig: 10
      save: True
      plot: True
      select:
        max_n_glitch: 10
        sig_glitch: 10

image

ahincks commented 3 months ago

Looks nice! Is the idea here that there's overlap with the stats being pushed to Grafana, via Tristan's system, such that if something looks off in the summary stats on Grafana, one can go over and look at these plots for more context? Or is the hope that people will regularly be reviewing the all the plots?

davidvng commented 3 months ago

Looks nice! Is the idea here that there's overlap with the stats being pushed to Grafana, via Tristan's system, such that if something looks off in the summary stats on Grafana, one can go over and look at these plots for more context? Or is the hope that people will regularly be reviewing the all the plots?

@ahincks Yes there will be overlap with the metrics published to Grafana. I think the idea is this should provide two methods for people to check data quality, one more quantitative and one more visual.

davidvng commented 2 months ago

Changed the plots for jumps and glitches: image

image

davidvng commented 2 months ago

@msilvafe this should be ready for review

msilvafe commented 2 months ago

@msilvafe this should be ready for review

I'll approve it, however after we talked and decided to switch the glitch + jump plots to these @mhasself expressed that he liked the old ones, perhaps you can bring them back or generate both.