Open bjhardcastle opened 1 day ago
The smarter thing in the long run is to stop trying to interpret the output from spikeinterface in the sorted data asset.
To do that we'd need to implement timestamp adjustment prior to sorting. which can already be done with np_codoecean. To make everything consistent, we'd also need to update existing raw data assets, then re-run sorting.
A good time to do this would be when (if?) switching to ks4.
pros:
cons:
@arjunsridhar12345 @alejoe91 In the past we've had occasional duplication of probes (e.g. probe A recorded on both Record Node 101
and Record Node 102
) - when we looked at the sorting pipeline's NWB file in the past, the units from both copies were pooled together, because they shared the same device attributes. Did that ever get fixed?
Was it intended to record the same probe on multiple Record Nodes?
I think I could add a safeguard mechanism in case this happens. If we detect that two device names are the same, then we can prepend the record node, so that the group name will become something: RecordNode101#ProbeA
What do yout hink?
Was it intended to record the same probe on multiple Record Nodes?
No, it only happens if we overlook something in the Open Ephys configuration.
If we detect that two device names are the same, then we can prepend the record node
I don't love the idea of introducing different naming conventions right at the end of the pipeline.. we'll inevitably look up ProbeA
in the units table, find nothing, and infer that it wasn't sorted.
Ideally, IMO, there would be a check during dispatch or pre-processing to see if the same device was recorded on multiple record nodes with the same configuration, then only sort one of the copies. We have this as a filtering step before upload now, so it shouldn't happen again for our sessions - but we do have some existing sessions on S3 with this problem, and it can happen fairly easily to others.
That's a good idea. I'll update the job dispatch capsule to skip such duplicate cases.
Organization of data in sorted assets has changed yet again and broken our NWB packaging.
We currently only have one session that needs to be re-sorted for the paper 1 datacube, but we'll likely find others with missing probes after QC review.
The easiest fix would be to use an older version of the pipeline, which isn't something we can customize in the production pipeline, but should be possible by duplicating and editing the nextflow files.
For now, any sorted assets created after September 24th will now fail in npc_ephys, similar to this: