Tests/Checks for quality control

[jerlich]

At the July 28th meeting, we discussed QC. A brief summary (of my memory) of the issues:

• I suggested defining APIs for session level test functions - e.g. take a sessid and return an DataFrame with cols: start, end, errorid, so that as new ideas for tests are developed, they are easy to add.
• We would want checks at several points: pre-ingestion, and several places along the processing pipeline.

[jkbhagatio]

I think there are two types of QC: one that operates online on the raw data, and the second that can operate offline on the data that is already stored on ceph.

I moved this issue to this acquisition repo because the more I think about this the more it makes sense have the bulk of the QC be the former, run online on the acquisition machine.

I think both types of QC can be written in any language, but output a common format, which can be the DataFrame/table suggested - maybe makes sense to write everything out in CSVs that are saved to /ceph/aeon/aeon/preprocessed/qc/online and /ceph/aeon/aeon/preprocessed/qc/offline ?

Both types could be run as system jobs. I think it makes sense to have all the QC code live in this repo?

[ttngu207]

I agree that there are generally two types of QC, online and offline as Goncalo suggested. For the offline part, based on our July 28th meeting, I've prototyped a set of DJ tables to specify the flow of this QC process.

Roughly, there will be multiple QC routines, checking for different aspects of the data. Each QC routine will have a corresponding function - the code live either in aeon_acquisition or aeon_mecha. If in aeon_acquisition, then the aeon_mecha repo must have aeon_acquisition repo as a dependency - i.e. must pip install the aeon_acquisition for DJ pipeline to run the QC part)

But all QC functions should returns i) a qc status code, ii) a comment/description (optional) and iii) a list of bad periods. (here, I'm thinking some offline QC routines could be loading the online QC results from the CSV file, to also store the online QC results)

In the diagram above, the table SessionQC is a computed table that, for a given session, executes all QC routines, calling the QC functions and storing the status codes and bad periods.

Downstream, the table GoodSession is another computed table that aggregates the results from all QC routines for a particular session, and determines if it is a Good or Bad session. (here, if we know ahead of time a session is bad, then the experimenter can just manually flag that session by inserting into the BadSession table.

Only GoodSession will proceed to any further analyses, and users can restrict to the GoodSession table for analyses of their own.

[Dario55]

QUALITY CHECK.... CHECK LIST I have listed below some failure mode we have experienced or we are very likely to experience and we should include in the QC list. Please add more. For some of them I have suggested some possible solutions

Session is bad, remove it:

• [ ] we initialise the wrong parameters at the start of the session and we stop and restart the session immediately It can be identified from the length, usually they last few seconds (Thinh can get set of manually labelled bad sessions from experimenters and have a table of this)
• [ ] the mouse behave in a strange way for reason X(sick or not food deprived enough) User needs to manually flag the session during the experiment, maybe using the annotation tool
• [ ] mouse did not get enough pellets to characterize the behaviour (e.g., less than 20-30, we have to decide a cut off) It can be done automatically during postprocessing
• [ ] There is a severe hardware failure (run out of pellets, FED3 jammed, all cameras stopped working) at some point in the session but we notice that it happened only after a while and a large portion of the session may be affected. Maybe user has to flag the session manually or can be automated

Some periods in the session are bad, identify and remove(?)/flag them:

• [ ] mouse is removed from the arena BEFORE we pressed stop: final part of the recording would contain us chasing the mouse and tracking errors. (Jai: catch experimenter in tracking, look for no objects (NaNs)) User needs to flag the session or maybe can be inferred from taking looking at tracking errors or area of tracked object
• [ ] Fed 3 got jammed. pellets stuck in the chute from the fed3 to the wheel. This causes a continuous interruption of the laser beam, so threshold is reached but no pellet is delivered. Fed3 gives zeros and never transition to 1 after threshold is crossed (or gives always ones and never goes to zero, cannot recall which way around). (Goncalo: consecutive same values in pellet trigger / detection; online)
• [ ] Fed3 runs out of pellets No beam break is detected after pellet delivery command is sent (Goncalo: consecutive same values in pellet trigger; online)
• [ ] tracking is lost. re-track the bad period of the session or flag the period if that's not possible If there are large jump in the tracking this is probably detectable automatically (Jai: look at video data for NaNs and large jumps in position tracking, come up with cases for different cases; need to see when picks NaNs vs. when picks other non-mouse objects)

Additional QCs

• [x] is everything temporally aligned correctly? (Goncalo: https://github.com/SainsburyWellcomeCentre/aeon_acquisition/issues/55)
• [x] did we drop frames? What offsets should be applied to re-sync? (Goncalo: dropped frame QC)
• [ ] was a given pellet consumed or not? It can be achieved with a sequence of conditions. If a threshold is reached and if pellet delivery command is sent and beam is broken then a pellet is very likely to be on the wheel. Mice stop spinning for few seconds after that a pellet is delivered, if they are eating it. If they don’t stop spinning, then the pellet will inevitably fell under the wheel. Also additional criteria based on mouse posture/motility in the post delivery time can be applied (Jai)
• [ ] Check that session was terminated by pressing end. _If no end is present check the time of last wheel movement or when user gets in the arena to take the mouse (tracking errors). (Jai: hopefully should fall out of test code for position tracking, could look at last pellet delivery and end session here)
• [x] Check if a pellet was manually delivered pressing shift+f (Goncalo: Bonsai key logger, or look at beambreak event that does not coincide with wheel threshold crossing)
• [x] Check if a pellet was manually delivered by user spinning the wheel (look at position data during pellet delivery: sessions: 8/9/2021 | BAA-1099790; 8/9/2021 | BAA-1099791)
• [ ] Weight is missing check from excel lab notebook or from previous session if a short session of same mouse same day proceeded the current one. If not available put nan (Jai: add to metadata code in API)

[jkbhagatio]

@glopesdev I "assigned" you to some of these issues in Dario's checklist, let me know if this is ok?

[glopesdev]

I think most of these have now been broken up into separate issues and resolved for Experiment 0.2. They are integral part of the online alert system and they are logged with standardized codes by the system.

The offline ingestion checks should probably be moved back into the aeon_mecha repo. It's possible that these offline checks have also been already broken down and replicated there, in which case we should probably close this issue altogether.

@jkbhagatio , @Dario55 , @ttngu207 any thoughts on this?

[glopesdev]

Closing as most of the scope of this issue has now been invalidated by either the alert system or offline QC routines. It will stay in the repo for reference.