Closed toni-neurosc closed 5 months ago
That's great! Thanks @toni-neurosc. Didn't know that such speed improvements would be possible even with those simple np functions!
That's great! Thanks @toni-neurosc. Didn't know that such speed improvements would be possible even with those simple np functions!
Well for 5 channels a 5x speed-up checks out, and the more channels you have in your experiment, the more speed-up in theory.
So I was looking at the profiling info, and I was quite happy that Bursts and Sharpwaves are now as fast as Hjorth. But after messing with Bursts and Sharpwaves for so many hours, I know that they're very complex features, so how come Hjorth is taking so long? Is it also super-complex?
Well it turns out no, it was just doing quite a lot of redundant work so I streamlined it a bit, which brought the computation time to 20% of the previous code, or a 5x speed-up.
Summary of changes:
diff
,var
andsqrt
calculations step-wise storing intermediate results to avoid repeating calculationsaxis
argument to run the calculations on the entire dataset at once, rather than 1 channel at a timenan_to_num()
only once at the very end, it's more efficiente and also safer (intermediatenan_to_num()
calls might result in divisions by zero, which result ininf
, whichnan_to_num()
will replace with the largest possible value for the correspondingdtype
)EDIT: did the same for LineLength, basically just applying the numpy functions directly to the 2D matrix, got a ~3x speed increase, although this feature was already very cheap.
EDIT2: just for reference, we're at this speed of computing for the default settings:
Total time to process 10.0s of data with 5 channels: 5.7076427936553955 seconds per run.
And I still have some aces up my sleeve, but I think the low hanging fruit has been more or less covered to I'm changing focus to GUI for now.