search

AllenNeuralDynamics / aind-watchdog-service

Data staging service that prepares acquistion data for cloud upload to Amazon S3 and Code Ocean processing.
https://allenneuraldynamics.github.io/aind-watchdog-service/
MIT License
2 stars 1 forks source link

Watchdog services completely hangs on OpenScope experiments #79

Closed jeromelecoq closed 1 month ago

jeromelecoq commented 1 month ago

Describe the bug At the end of every 2p experiment, the ophys team triggers a manifest file creation in watchdog service GUI. This GUI creates a manifest yml file onto the program data aind folder. See attached below for a good one. In the last month, data for the Learning mFish project successfully created their. All of OpenScope experiments were not able to reach this step. With OpenScope, the GUI just hangs in there forever with no feedbacks.

To Reproduce Steps to reproduce the behavior:

  1. Launch aind-watchdoc-service.exe on Mescope
  2. Enter user and session id
  3. Click submit

Expected behavior A manifest file should be created in "program data/aind/watchdog.... (See example below) No ERROR are present in "aind-watchdog-service.log No ERROR are present In prepare_transfer.log

A good exp should show this: 

2024-10-10 16:05:07,164 - root - INFO - User name: ahrial young, Session ID: 1399309905
2024-10-10 16:05:07,237 - root - INFO - Project ID: LearningmFISHTask1A, Subject ID: 747667
2024-10-10 16:05:07,358 - root - INFO - Generating user settings
2024-10-10 16:05:07,365 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/ecephys_sessions.json?id=1399309905
2024-10-10 16:05:07,392 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/behavior_sessions.json?id=1399309905
2024-10-10 16:05:07,410 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/ecephys_sessions.json?id=1399309905
2024-10-10 16:05:07,429 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/behavior_sessions.json?id=1399309905
2024-10-10 16:05:07,453 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/ecephys_sessions.json?id=1399309905
2024-10-10 16:05:07,473 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/behavior_sessions.json?id=1399309905
2024-10-10 16:05:15,801 - root - INFO - Project ID LearningmFISHTask1A
2024-10-10 16:05:15,804 - root - INFO - Generating manifest file

Our OpenScope exp shows this : 

2024-10-11 10:35:06,088 - root - INFO - Submit button clicked
2024-10-11 10:35:06,096 - root - INFO - User name: shiella caldejon, Session ID: 1399431639
2024-10-11 10:35:06,180 - root - INFO - Project ID: OpenScopeTexture, Subject ID: 740066
2024-10-11 10:35:06,333 - root - INFO - Generating user settings
2024-10-11 10:35:06,354 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/ecephys_sessions.json?id=1399431639
2024-10-11 10:35:06,372 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/behavior_sessions.json?id=1399431639
2024-10-11 10:35:06,416 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/ecephys_sessions.json?id=1399431639
2024-10-11 10:35:06,434 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/behavior_sessions.json?id=1399431639
2024-10-11 10:35:06,549 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/ecephys_sessions.json?id=1399431639
2024-10-11 10:35:06,565 - np_session.databases.lims2 - DEBUG - Requesting http://lims2/behavior_sessions.json?id=1399431639

Desktop (please complete the following information):

Additional context See example manifest file for Learning mFish that is good. manifest_20241010160516.txt

see GUI log prepare_transfer.log

jeromelecoq commented 1 month ago

Also note that http://lims2/behavior_sessions.json?id=1399431639 and http://lims2/ecephys_sessions.json?id=1399431639 are returning empty lists. Obviously those are NOT ecephys experiment so is that expected?

arielleleon commented 1 month ago

This is b/c the stimulus epoch and stim table generation are using ephys utilities. This will be resolved with this ticket; https://github.com/AllenNeuralDynamics/aind-physio-arch/issues/52 where we consolidate the aind-metadata-mapper library