search

Eden-Kramer-Lab / spectral_connectivity

Frequency domain estimation and functional and directed connectivity analysis tools for electrophysiological data
GNU General Public License v3.0
122 stars 44 forks source link

Speed/Memory Bottlenecks #6

Open edeno opened 7 years ago

edeno commented 7 years ago
Who8MyLunch commented 7 years ago

Hi @edeno you might consider using FFTW (https://github.com/pyFFTW/pyFFTW) if you're looking for faster performance. It makes a huge improvement. It's a drop-in replacement for Numpy and Scipy FFT implementations.

edeno commented 7 years ago

Thanks @Who8MyLunch. I will definitely check that out.

You don't happen to have any suggestions for something that has to take the dot product many times do you? I'm considering moving that portion to using cython, but I don't have much experience with that.

Who8MyLunch commented 7 years ago

@edeno there's a number of options for accelerating your dot products, the simplest being Numpy and the next being Cython. Cython is good, but it's not trivial. Numpy is trivial to use. There are many examples around the internet of using both to speed up fundamental work tasks like dot products. Good luck!

christian-oreilly commented 3 years ago

I had to delete any use of lru_cache in my local copy to avoid the memory exploding to unreasonable values (above 64 GB). Even deletion of the objects and explicit garbage collection would not do the trick. So, bottom line, depending on where your use case falls with respect to the speed/memory usage tradeoff, caching might not always be a good idea. A possible improvement would be to make the caching optional and control it by an argument to the function using it (set by default to True for backward compatibility).

edeno commented 3 years ago

@christian-oreilly That makes sense. I'll think about a better way to handle this. Thank you for that feedback.