Very slow loading neuropixel binary data

[mikailweston]

Error with concurrent plotting of data that overlaps in time

By default all NP data is acquired at the same time: multiple LF and AP files per folder, 1 each for each probe.

If there is more than one file in the folder PyEcog plots all data from the same folder concurrently even if only one file is selected in the file tree

300s to display 30s of 385 channel data from 4 files (2 x LF and 2 x AP)

Work around: put each file in a separate folder

Data loading still slow

• LF data: 25s for 385 channels (sample rate ~2500 Hz)
• AP data: 120s for 385 channels (sample rat ~30,000 Hz)

@mfpleite Would it be possible to plot down sampled array as a whole rather than each channel individually?

[mfpleite]

@mfpleite Would it be possible to plot down sampled array as a whole rather than each channel individually? I don't understand what you mean

Regarding multiple concurrent files: that is the original intended behavior to cope with chopped telemetry data. We didn't expect multiple files of the same sort overlapping (on purpose) in time. We can think about implementing this, but it would be a new feature.

To speed up the plots we can impose a maximum data size to be displayed at the start - any suggestions on what this size might be?

[mfpleite]

when opening a folder we can make each file a different animal in the temporary project if that helps...

[mfpleite]

when opening a folder we can make each file a different animal in the temporary project if that helps... I have made this change in the last comit

[mikailweston]

Thanks. I have shared a 27GB AP file and 2GB corresponding LF file.

I think the main issue with the speed of display is computing the scale for each channel in the overview, and then drawing each pen + doing the same for inset view.

It would be helpful to limit the amount of data viewed in the overview plot for NP data, for example a few minutes.

Would a way of speeding this up be to compute a downsampled and scaled numpy array and saving this in a buffer before drawing the overiew pens. The insetview could be limited to 30s of data that hasn't been downsampled?

[mikailweston]

Also would it help to turn off autorange in the view box until after plotting?

https://stackoverflow.com/questions/17103698/plotting-large-arrays-in-pyqtgraph/17108463

[mfpleite]

Also would it help to turn off autorange in the view box until after plotting?

It is already disabled in lines 113 and 114 of paired_graphics_view.py. The thread might have good tips, I'll have a look.

[mfpleite]

Ok the decimation closes this for the time being :)