ephys upload planning

[alexpiet]

@XX-Yin and @KanghoonJ have asked about using the aind-watchdog service to upload data to AWS. This brings up questions about how we want upload and processing to work

• [ ] does aind-watchdog upload ephys data? Or just behavior data
• [ ] Do all ephys rigs want the same upload process?
• [ ] What pipeline gets triggered on upload?
• [ ] Pipeline coordination across modalities: #678

[XX-Yin]

@alexpiet Thanks for this organizing this.

1. The ephys data is more complex, and I think we can do it separately not using the watchdog service.
2. The data location in the scratch is different.
3. It's the same as in the behavior rig if we don't upload the ephys data.

[alexpiet]

To clarify:

1. You want to transfer the behavior data, but not the ephys data. is the ephys data stored in the same location as the behavior data on the acquisition computer? Or is it elsewhere?
2. You want the behavior data from ephys sessions transferred to a different location on VAST compared to behavior only sessions. Why does this matter if we are uploading ephys data separately?
3. Ok. This is probably the easiest solution. We will generate a behavior only NWB file. And its up to the ephys pipeline to later merge in any ephys processing.

[XX-Yin]

1. Yes, only to transfer the behavior data. The ephys data is stored in a different computer.
2. The behavior data can go to the same VAST location no matter it's ephys session or only behavior session.

[alexpiet]

Ok, then I think we should be able to move forward with:

1. Installing the aind-watchdog-service on the ephys computer
2. In ForagingSettings.json set AutomaticUpload to True

Each ephys rig can opt in as they want

[alexpiet]

Proposed upload strategy for behavior data on ephys computers

1. GUI creates an upload manifest for aind-watchdog-service by setting 'AutomaticUpload' to True in ForagingSettings.json
2. In the evening behavior data for each session is transfered to VAST and triggers upload to AWS by the aind-watchdog-service
3. Upload triggers the behavior processing pipeline which creates a behavior only NWB file.
4. Ephys data is uploaded elsewhere, and creates its own NWB file
5. behavior NWB and ephys NWB are merged into a final NWB file.

[XX-Yin]

Proposed upload strategy for behavior data on ephys computers

1. GUI creates an upload manifest for aind-watchdog-service by setting 'AutomaticUpload' to True in ForagingSettings.json
2. In the evening behavior data for each session is transfered to VAST and triggers upload to AWS by the aind-watchdog-service
3. Upload triggers the behavior processing pipeline which creates a behavior only NWB file.
4. Ephys data is uploaded elsewhere, and creates its own NWB file
5. behavior NWB and ephys NWB are merged into a final NWB file.

This sounds good!

@alexpiet Do we have a pipeline to upload data for old sessions? These old sessions have been copied to the VAST.

[alexpiet]

@XX-Yin We do not have any infrastructure for uploading old data

[XX-Yin]

@XX-Yin We do not have any infrastructure for uploading old data

@alexpiet Could we plan this? I think we need to modify the manifest file.