drasmuss
commented
3 years ago You could definitely have your spectrogram processed one step at a time by passing NengoDL data with shape (batch, time, frequency). But it would be a very different kind of model than the one you have now (since you're no longer processing spectogram "images" that are filtered by Conv2D layers). It would be an open research problem, based on your particular application, what the best way is to go about doing that, but using an LMU is probably a good option!
You might be interested in the forum which has more people than can help with these kinds of research-based questions.
loic001
Hello,
I converted a ConvNet to a SNN and it works.
With the ConvNet, the input format is
[Batch, Channels, Height*Width]. In my particular case, it is[Batch, 1, Frequency*Time]because I pass spectrograms as input (the first layer reshape to[Batch, 1, Frequency, Time]before passing it to the first convolution layer). There are only convolutions (no recurrent connections, no RNN/LSTM..., and therefore no sequence/timestep).Once the conversion to SNN is done, the input format with NengoDL is
[Batch, Steps, NodeSize], i.e. in my case[Batch, 1, Frequency*Time].As a result, like ConvNet, the spectrogram is eaten at once, and not over time (Steps=1). Do you think it is possible to design a SNN with Nengo which takes the spectrogram over time ? I would like a SNN that deals with something like this [Batch, Time, Frequency] (Steps=Time). I need this because it's more biologically plausible.
Is trying to convert a LSTM to a SNN is a solution ? Legendre Memory Units would be even better ?
Thank you for your help :+1: