Upload split recordings

AllenInstitute / npc_lims

Tools to fetch and update paths, metadata and state for Mindscope Neuropixels sessions, in the cloud.

https://alleninstitute.github.io/npc_lims/

MIT License

0 stars 0 forks source link

Open bjhardcastle opened 10 months ago

bjhardcastle commented 10 months ago

[x] rearrange folders before upload
- in experiment1 folder, make consecutive recordingN folders
[ ] adjust timestamps / sync messages as necessary, to appear as if all recordings from one session
[ ] upload without sorting, if possible
[ ] manually trigger sorting with CONCAT=True
[ ] keep a record of "unobserved" interval (add via pynwb.misc.Units.add_unit(obs_intervals=...) and use with nwb.get_unit_spike_times() + nwb.get_unit_obs_intervals() )

bjhardcastle commented 10 months ago

confirm method works, then upload affected sessions:

[ ] '2023-07-25 670180'
[ ] '2023-07-20 670181'
[ ] '2022-09-20 628801' watch out: videos recorded in at least 2 parts, before upload need to:
- [ ] find possibly missing 3rd part
- [ ] see if 1st videos are recoverable (the don't open currently)
- [ ] find missing info jsons
- [ ] concat video frames or deal with multiple blocks of frames on sync

bjhardcastle commented 7 months ago

Needs a processing script prior to upload for recordings resumed after software restart:

samples need and timestamps need to appear under the same global clock
adjust sync_messages.txt, sample_numbers.npy, timestamps.npy, and potentially other files, based on time separation between recordings https://open-ephys.github.io/gui-docs/User-Manual/Recording-data/Binary-format.html
separation between recordings could be found to nearest 1 ms from sync_messages.txt - but it should be possible to find it accurately with barcodes and sync.h5

bjhardcastle commented 7 months ago

Some sessions were uploaded without sample/timestamp adjustment - will need to reupload all