search

fangwei123456 / spikingjelly

SpikingJelly is an open-source deep learning framework for Spiking Neural Network (SNN) based on PyTorch.
https://spikingjelly.readthedocs.io
Other
1.29k stars 236 forks source link

Is it possible to use the raw event stream (Neuromorphic dataset) to train an SNN model without using frame reconstruction? #404

Open djessy1998 opened 1 year ago

djessy1998 commented 1 year ago

Issue type

SpikingJelly version

0.0.0.0.14

Description

Hi! First of all, thanks for your work. I have a question related to training a simple SNN, for example. I'm using a neuromorphic dataset where I would like to train my model using the event stream directly, asynchronously, without frame reconstruction. I don't think I've seen something like this in your examples.

Question: Is it possible to do this with SpikingJelly?

Also, for classification on DVS128Gesture, could the results obtained using the raw event stream be radically different from the results with frame reconstruction?

Thank you for your answers!

fangwei123456 commented 1 year ago

Is it possible to do this with SpikingJelly?

Yes, you can convert the (x, y, t, p) to the tensor format, and send these events to the SNN.

But the number of time steps will be in millions, and it seems impossible to train the SNN. Thus, the downampling on time is necessary.