search

Xingrun-Xing / SpikeLM

This is the implentation of our paper "SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms" in ICML 2024.
16 stars 0 forks source link

Amazing work！When will the code be open sourced？ #2

Open zxt6174 opened 3 months ago

Xingrun-Xing commented 3 months ago

Thanks for reading our paper. We will release the code before the conference, maybe very recently.

zxt6174 commented 2 months ago

Congratulations on your acceptance of your paper.！But while reading the paper, I had some questions. Spikingformer was originally used for image classification tasks, what modifications do I need to make to use it for NLP tasks? Is the training method used direct training, or is it pre-training + fine-tuning like Spikebert? It doesn't seem to be stated in the paper。

Xingrun-Xing2 commented 1 month ago

Congratulations on your acceptance of your paper.！But while reading the paper, I had some questions. Spikingformer was originally used for image classification tasks, what modifications do I need to make to use it for NLP tasks? Is the training method used direct training, or is it pre-training + fine-tuning like Spikebert? It doesn't seem to be stated in the paper。

Hi, sorry for the late reply. We use the Spikingformer as a comparison with the same training scheme as SpikeLM, including the same pertaining and fine-tuning. To make a fair comparison, we also remove the spiking neuron in the value of self-attention (as the caption in Table 3).

If you want to train spikingformer in NLP tasks, it is fine to replace the transformer block defined in spikeLM-BERT/spike_bert.py.

zxt6174 commented 1 month ago

Sorry, I don't understand very well, if I want to use SpikingFormer for language tasks, what should I change in the source code of SpikingFormer, or can you upload a modified "spikingformer_model.py" ?