Create Visual Behavior Ecephys project class (cloud-based)

[wbwakeman]

This is a project-level class that is used by public AllenSDK users to get metadata about all sessions. This allows them to filter and retrieve specific files of interest to them.

Need to create a Visual Behavior Ecephys project class. This will analogous to the VBO cache class implemented in

https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/brain_observatory/behavior/behavior_project_cache/behavior_project_cache.py (modern - Sprint 2021)

Implement a project class that provides methods to:

• download session NWB files (LFP data interactions will be part of a later release)
• instantiate a DataObject
• retrieve metadata files

We are only supporting a from_s3 API. There will be no from_lims API for this data release.

Tasks

• [x] Follow instructions in informatics_data_release_tools repo to create a sandbox bucket for cartoon data release

Instructions to create bucket https://github.com/AllenInstitute/informatics_data_release_tools/tree/main/deploy

Instructions to upload data https://github.com/AllenInstitute/informatics_data_release_tools

• [x] Upload cartoon data release to sandbox S3 bucket
• [ ] Write behavior cache class that downloads data and metadata from sandbox S3 bucket

Validation criteria:

• [x] cache class needs a from_s3_cache method like the VBO cache class (VisualBehaviorOphysProjectCache) to instantiate the class with a from_s3 API that points to the bucket being used for the release.
• [x] cache class needs get_ecephys_session method to download NWB files from S3 and instantiate them into BehaviorEcephysSession objects.
• [x] cache needs getter methods for all of the metadata tables that will be uploaded to this S3 bucket. Cartoon metadata files can be found in

  /allen/aibs/informatics/danielsf/ecephys_table_examples/realistic

  Note: they will not include local file paths or unique file ID column.

• [x] getter methods return expected objects based on S3 uploads.

[danielsf]

There does appear to be an existing implementation of this class using just the from_lims and warehouse APIs

https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/brain_observatory/ecephys/ecephys_project_cache.py

[danielsf]

32 should probably be completed before code is actually written for this ticket

[danielsf]

Metadata tables we need to be able to get are:

units.csv
channels.csv
probes.csv
sessions.csv
behavior_sessions.csv

These should all be such that they can be read in with pandas.read_csv(), yielding a DataFrame.

For now: NWB files will only be looked up from sessions.csv (because we aren't releasing behavior sessions yet). The column for file ID will be called file_id.