re-scope VBN release

[danielsf]

We need to discuss with the scientists what our plan is for the VBN release so that we understand the scope of work that we are responsible for.

Questions to answer

• [x] What ecephys sessions need to be released (get a list of ecephys_session_ids)?
• [ ] What behavior sessions need to be released, assuming we decide the behavior sessions are in-scope (get a list of behavior_session_ids)?
• [ ] What files need to be released
  • [x] Ecephys experiment NWBs?
  • [ ] Behavior only NWBs?
  • [ ] LFP NWBs?
• [ ] What fields need to be contained in each set of NWB files (e.g. is behavior tracking data in scope for this release)?
• [ ] What metadata tables need to be released?
  • [ ] What columns should they contain?
• [ ] What is the process for validating the contents of NWB files?
  • [ ] Are there scripts that Pika can just run without having to continuously communicate with the scientists?
• [ ] What is the process for validating the contents of the metadata files?

Tasks

• [ ] Once these questions have been answered, Pika should estimate how much time we require to complete all of the tasks.

Validation

• [ ] A final list of ecephys_session_ids and behavior_session_ids is recorded in this ticket
• [ ] A final list of NWB files to be released and the fields they need to maintain is recorded in this ticket
• [ ] A final list of metadata files and their schemas are recorded in this ticket
• [ ] The process for data and metadata validation is recorded in this ticket

[danielsf]

On 3/28/2022 Corbett emailed us a list of in-scope sessions for release. They are listed in this spreadsheet

The text of Corbett's email is

I've compiled a list of the experiments for release: VBN_release_shared.xlsx This totals 154 sessions (with TRUE in the "Release" column), a subset of the 180 I originally gave Wayne. Note that for some of these experiments, individual probes should be excluded (if they exist in LIMS to begin with). This is indicated in the 'Bad probes' column. I've also left two columns to note which experiments will be flagged for OPT or ephys abnormalities, which will be flagged in the metadata tables. I'll work on finalizing these labels over the next week or so.

The annotation data for these experiments is finished, and can be found here (same place as before):

\allen\programs\braintv\workgroups\neuralcoding\corbettb\VBN_production

[danielsf]

See #32 for discussion of the contents of the metadata files

[danielsf]

We did discuss this with the scientists. The decisions were:

• only releasing the spike data NWB files for the ecephys sessions Corbett has designated (there will be subsequent releases to include behavior-only and LFP data)
• Regarding contents of NWB files: we will implement what we have inferred from the code we have, deliver the draft NWB files and see if the scientists ask for something to be added.
• Process for validation: we will generate a draft data/metadata set, hand it off to the scientists, respond to their feedback, and iterate.

For what it's worth: the spreadsheet where Nathan tried to document the contents of the NWB files is here

https://alleninstitute-my.sharepoint.com/:x:/g/personal/nathan_sjoquist_alleninstitute_org/EWQAQtWBmDhCiQipTSOPq3sBrWhpG67xgcL1MMuRrzP5uA?e=IFyZ4P&CID=47377fdf-8906-9a1b-6c31-33a5e9d9593a