Create upload manifest.yml file

[alexpiet]

Pull Request instructions:

• Please follow the update protocol
• Answer the questions below in detail. Your responses will be emailed to experimenters.
• If the experimenters must do anything new, provide detailed step by step instructions on the wiki
• If computer maintainers need to manually update anything, provide detailed step by step instructions
• Use markdown syntax in order for your comments to be rendered reliably in the email: "1." instead of "1)", use four spaces for indents.
• If you use the keyword "skip email" in the title, it will skip the email updates
• Merges from "develop" into "production_testing" should use the keyword "production merge" in the title for reliable indexing of updates
• Merges from "production_testing" into "main" should use the keyword "update main"

Describe changes:

• Generates the manifest.yml file that triggers aind-watchdog-service
• set upload times to 11:59pm for all sessions
• Adds a setting in ForagingSettings.json called AutomaticUpload, which defaults to True. To disable automatic upload (EPHYS rigs may want to do this), then set this to False.

What issues or discussions does this update address?

• resolves #556

Describe the expected change in behavior from the perspective of the experimenter

• If the manifest generation fails, users will get an alert that the data will not be automatically upload
  • IMPORTANT: if this happens, alert the owner of the mouse.

Describe any manual update steps for task computers

• Determine if you want to opt-out of automatic uploads, and if so, set ForagingSettings.json/AutomaticUpload to False
  • [ ] Ephys1 @KanghoonJ
  • [ ] Ephys3 @XX-Yin
• Configure watchdog service (steps below)
  • [ ] Ephys1 @KanghoonJ
  • [ ] Ephsy3 @XX-Yin
  • [ ] 428 @hagikent
  • [ ] 446/447 @hanhou @alexpiet
• Install aind-watchdog-service on each rig computer, and enable TaskScheduler to automatically start the service
• Add a watch_dog.yml file to ~/Documents/aind_watchdog_service
• Set the WATCH_CONFIG system variable to point to ~/Documents/aind_watchdog_service
• watch_dog.yml should contain:

  flag_dir: {~}/Documents/aind_watchdog_service/manifest
  manifest_complete: {~}/Documents/aind_watchdog_service/manifest_completed
  webhook_url: null

Was this update tested in 446/447?

• [x] tested in 447

[alexpiet]

Current manifest test output

acquisition_datetime: '2024-06-27 11:30:49'
capsule_id: c089614a-347e-4696-b17e-86980bb782c
destination: //allen/aind/scratch/dynamic_foraging_rig_transfer
modalities:
  behavior: C:/behavior_data/446-7-D/0/behavior_0_2024-06-27_11-30-49/behavior
  behavior-videos: C:/behavior_data/446-7-D/0/behavior_0_2024-06-27_11-30-49/behavior-videos
  fib: C:/behavior_data/446-7-D/0/behavior_0_2024-06-27_11-30-49/fib
mount: behavior_0_2024-06-27_11-30-49
name: behavior_0_2024-06-27_11-30-49
platform: behavior
processor_full_name: AIND Behavior Team
project_name: Behavior Platform
s3_bucket: private
schedule_time: '2024-06-27 23:59:00'
schemas:
- C:/behavior_data/446-7-D/0/behavior_0_2024-06-27_11-30-49/metadata-dir/session.json
- C:/behavior_data/446-7-D/0/behavior_0_2024-06-27_11-30-49/metadata-dir/rig.json
script: {}
subject_id: 0

[alexpiet]

@hagikent @hanhou @XX-Yin

This is ready for review.

[hagikent]

PR seems to be ready for review but we are not ready to review...! As we are still figuring out how to set mount at watchdog properly (i.e. not fully confirmed if mount will be input_data_mount when translated to transfer-service), let's wait for a while. Also, s3_bucket: private may be susceptible to change...?

[hanhou]

s3_bucket: private may be susceptible to change...?

Let's still keep it as private

[hagikent]

@alexpiet Just comparing with manifest.yml files produced by Han, the following two entries have quotations. I don't know if the downstream process is agnostic to this or not.

acquisition_datetime: '2024-06-27 11:30:49' schedule_time: '2024-06-27 23:59:00'

[alexpiet]

@alexpiet Just comparing with manifest.yml files produced by Han, the following two entries have quotations. I don't know if the downstream process is agnostic to this or not.

acquisition_datetime: '2024-06-27 11:30:49' schedule_time: '2024-06-27 23:59:00'

Fixed