search

AllenInstitute / AllenSDK

code for reading and processing Allen Institute for Brain Science data
https://allensdk.readthedocs.io/en/latest/
Other
333 stars 149 forks source link

Upload data to S3 using the data release tool. #2597

Open morriscb opened 1 year ago

morriscb commented 1 year ago

Upload NWB files once Corbett okays them to visual-behavior-neuropixels-data. Likely to run into some issues as the structure has changed. Keep old version files as is.

Unsure how behavior only sessions and the LFP/probe data will behave here. May run into bugs.

danielsf commented 1 year ago

One thing to be careful of:

The re-organizing of the behavior_ecephys_session files (moving them from root into their own sub-directories by session_id) will cause the data release tool to run a remove operation on the nwb files that have already been uploaded to the S3 bucket.

I do not know what this will do to users who want to download data from VBN v1.0. It is possible that, after the aws rm operation has been run, it will be impossible to get the old files back, even though version control has been turned on for this bucket. You will probably want to test this operation (uploading files, removing them, and then retrieving the removed files with a CloudCache) in someone's S3 sandbox before proceeding with the upload.

Granted: since release 2.0 isn't going to actually change the behavior_ecephys_session files (right?), it may be okay to say "we no longer support v1.0; please use v2.0"

Feel free to reach out to me if my concern does not make any sense.