Need to create session.json schema

[alexpiet]

@dyf said I need to (1) generate a list of fields that we need to include, (2) figure out how to populate each field from the saved behavior data .json

assume d is the GUI created json file

• experimenter_full_name
  • d['Experimenter']
• session_start_time
  • d['Other_SessionStartTime']
• session_end_time
  • d['Other_CurrentTime']
• subject_id
  • d['ID']
• session_type
  • "dynamic-foraging-behavior"
  • should probably but stage/curriculum info here? (maybe lives in stimulus info)
• iacuc_protocol
  • "????"
  • I assume this is fixed
• protocol_id
  • This is for protocols.io
  • should be DOI for that protocol
  • For now, but N/A
• rig_id
  • d['box']
  • We have box names like 447-1-A, but I think rig_id is different
  • We should get this from rig.json
• animal_weight_post
  • d['WeightAfter'],

• data_streams

  • everything being recorded

  • for each stream: Start/end times, what devices are present:

  • stimulus_device_names: (will move to stimulus epochs)

  • mouse_platform_name: from rig.json

  • active_mouse_platform (bool): False

  • stream_modalities: which modalities are active

  • Calibrations

  • Water calibration, sometimes a laser calibration, embed each here

  • for each calibration: calibration_date, device_name, description, input, output, notes

  • Maintenance class?

  • reward_delivery:

  • lick spout stage: (need to indicate that its variable, the data is saved in NWB files) RewardSpoutConfig.variable_position

  • reward_consumed_total

  • reward_consumed_unit

  • stimulus_epoch

  • start/end time

  • stimulus classes: BehaviorStimulation (This is outdated, no longer BehaviorStimulation, check schema docs)

    • behavior_name, add curriculum name/stage/schema here
    • session_number: int # of sessions this mouse has done
    • behavior_software= list
      • name, version, url, parameters
    • output_parameters: # trials, # rewards

see docs here: aind-data-schema: src/aind_data_schema/models

• https://github.com/AllenNeuralDynamics/aind-data-schema/tree/main/src/aind_data_schema/models

[hanhou]

• https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/68

[hanhou]

Some code snippets from TransferToNWB may be helpful (most backward compatibility issues already handled) https://github.com/AllenNeuralDynamics/dynamic-foraging-task/blob/427a450c7ef45d5648406dc3ea22644d08fae2b6/src/foraging_gui/TransferToNWB.py#L152-L174

[alexpiet]

steps for creating session.json

• Collect data, could be across multiple computers (behavior and ephys, for example)
• Copy data to VAST, with expected organization by data modality
• Run script that creates session.json from data modality, saves in metadata directory
• Upload to AWS

[alexpiet]

Useful example scripts for making metadata files: https://github.com/AllenNeuralDynamics/CTLUT-metadata-gen/tree/main/src/CTLUT_metadata_gen

[hagikent]

Copy data to VAST, with expected organization by data modality Run script that creates session.json from data modality, saves in metadata directory Upload to AWS

Is that what you discussed with Saskia today? I thought GUI would directly generate session.json but considering the multimodalness of our data collection, this makes sense. Since "copy data to VAST" is at the moment done by a simple Robocopy routine, let's discuss if we need an active reorganization of data during this copy.

[alexpiet]

Copy data to VAST, with expected organization by data modality Run script that creates session.json from data modality, saves in metadata directory Upload to AWS

Is that what you discussed with Saskia today? I thought GUI would directly generate session.json but considering the multimodalness of our data collection, this makes sense. Since "copy data to VAST" is at the moment done by a simple Robocopy routine, let's discuss if we need an active reorganization of data during this copy.

Yes, this is what we discussed as a solution to multi-modal issues. I think as long as the data copying to VAST keeps the organization that the rig computers save out, then that is fine.

@XX-Yin is going to create a draft script that generates session.json. For FIP/Ephys sessions we will need to modify the script to include additional information

[hagikent]

Ok, thanks. I already have ophys_session.json for my other photometry based platform, and just by extracting some parts would be sufficient to make the FIP-piece of session.json to be merged into at the level of VAST.

as long as the data copying to VAST keeps the organization that the rig computers save out

Yes, that's the case. \allen\aind\svc_aind_behavior_transfer \allen\aind\svc_aind_FIP_transfer

[alexpiet]

resolved by #405