search

brian-team / brian2cuda

A brian2 extension to simulate spiking neural networks on GPUs
https://brian2cuda.readthedocs.io/
GNU General Public License v3.0
60 stars 12 forks source link

Brian2Hears and Brian2CUDA #308

Closed slyyyle closed 1 year ago

slyyyle commented 1 year ago

Hello all. I have been using brian2hears EXTENSIVELY over the past 8 months for my thesis. I have rather fallen in love with this package, as it allows me to use more complex filterbanks such as the linear and logarithmic gammachirp to construct custom image representations for neural networks. I've found the linear gammachirp works REALLY well across many classification models.

That being said, and also mentioned in a previous issue, is that there seems to be no CUDA functionality for brian2hears. As of now, I am simply generating the image representations and saving them off into a flat pandas dataframe, and then using that dataframe to create my batches. This is really really time intensive for many models.

I wanted to start a discussion about the art of the possible. I firmly believe that if we were able to parallelize or speed up the computation for some of these cochlear models, we could then designate certain arguments for the image representation as custom PyTorch trainable parameters (glide slope, time constants, etc) to allow a neural network to DICTATE how it creates its own inputs to best classify a sound. That being said, it seems as if there's already been extensive effort using .process() to parallelize the computations. I imagine this could be done on a class by class basis with some clever loss function coding, however this is just so theoretical at this point and CURRENTLY I think the time it takes to preprocess the data would cause a training loop to take multiple days.

Could I get an informed opinion from one of you devs on the art of the possible in this area? I know it won't happen anytime soon, but as I progress into Computer Vision I could see myself learning the pre-requisite skills necessary to make real contributions like this. I strongly believe that brian2hears could be a very accessible gateway for python users in DSP and audio classification.

slyyyle commented 1 year ago

Conversely, I was thinking of implementing similar filterbank preprocessing in PyTorch, but I believe I would run into issues as it seems like you have done extensive work to reduce computational complexity of these models already via .process(). I have no comparison point via computation time as I've only really used brian2hears.

denisalevi commented 1 year ago

Hi @slyyyle,

brian2 is a simulator for spiking neural networks. brian2cuda implements a backend for the brian2 simulator, such that any model that supports cpp_standalone in brian2, can also be simulated on GPUs with brian2cuda.

brian2hears is an auditory modeling toolbox that uses brian2. But unfortunately, it does not support cpp_standalone as @mstimberg mentioned in your previous issue. If you want brian2hears models to run on the GPU with brian2cuda, you would need to adapt those models to be compatible with cpp_standalone, then brian2cuda should work out of the box. Hence, this issue is better placed in the brian2hears repository: https://github.com/brian-team/brian2hears (even though that project is kind of orphaned as @mstimberg said, so it might be up to you if you want to dig into it).

That being said, I have the impression you are not simulating spiking neural networks but rather are using the brian2hears utility functions to generate filters etc. (I have no experience with brian2hears, so I'm just guessing here). If you want those functions to be implemented on GPU, you will have to implement them yourself, e.g. with libraries that support GPU execution, either directly in machine learning libraries such as pytorch, or using libraries for "Numpy on GPU", e.g. CuPy. Again, this would be an extension for the brian2hears package, and development should therefore happen in that repository.

All of this has nothing to do with brian2cuda though. So I don't think I can help you here.

slyyyle commented 1 year ago

@denisalevi You are correct in that I am not simulating spiking neural networks, but instead utilizing the filterbank functions primarily. Thank you for making the distinction between brian2, brian2cuda, and brian2hears, it makes sense that this issue is probably better posted on the brian2hears repo - my apologies!