Mindscope informatics processing support

[wbwakeman]

http://confluence.corp.alleninstitute.org/display/IT/Informatics+processing+support

As part of this work, you may consider entering a new issue(s) for a way to automate/improve these tasks.

This operations task is time-boxed. No more than approximately one hour per day should be spent on this. Create a new issue for tasks that exceed that, to be added to the backlog and addressed in some future sprint.

Acceptance criteria:

• [x] Product Owner assesses that BrainTV operations teams are satisfied that data processing happening consistently and successfully, and that any outstanding issues are being addressed

[djkapner]

@danielsf making some notes here.

• [x] 2020-12-08 : 3 failures in non-nway queues

  • 1047889126 09/02/2020 05:02:11 09/16/2020 08:42:18 09/16/2020 08:47:22 5 BEHAVIOR_OPHYS_WRITE_NWB_QUEUE this one has been around for a while. Related to https://github.com/AllenInstitute/AllenSDK/issues/1729 . We are still waiting for MPE. Leave here as reminder. There is also 1 entry in the SQL query, which refers to this same failure.
  • 1067671569 12/03/2020 00:24:25 12/04/2020 13:03:51 12/04/2020 13:07:35 4 MESOSCOPE_FILE_SPLITTING_QUEUE Nick made a ticket for it last week: https://github.com/AllenInstitute/ophys_etl_pipelines/issues/123
  • 1068194488 12/04/2020 23:19:27 12/04/2020 23:37:29 12/04/2020 23:50:47 13 OPEN_OPHYS_MOTION_CORRECTION_QUEUE moved to DEBUG queue and archived.

  NWAY matching has 21 containers failing from today (12/08). This is expected as part of QC-a-thon data survey

• [x] 2020-12-09: nothing new since yesterday.
• [x] 2020-12-10: nothing new since yesterday.
• [x] 2020-12-11: nothing new since yesterday.
• [x] 2020-12-14:
  • after email from Taylor re: 1067671569, archived this failure.
  • archived some experiment that was a test, showing a glass reticule.
• [x] 2020-12-15: nothing new

[wbwakeman]

I think that one is still waiting on MPE