SpikeGPT

[thistleknot]

Feature request

Extract the spiking nature of the LLM and port that [set] of features over for training/inference,.

https://github.com/ridgerchu/SpikeGPT

Motivation

the benefits would result in more efficient computational costs (x22 reduction).

Your contribution

I am willing to test, trace down bugs, and push. I'm still new in the world of llm backend coding.

[amyeroberts]

Hi @thistleknot, thanks for opening this feature request.

Just skimming the repo, my understanding is that SpikeGPT already has a set of pretrained weights available.

If you (or someone else) would like to make this model available through the transformers API, the easiest and fastest way is to add it directly on the hub - here's a guide: https://huggingface.co/docs/transformers/custom_models.

[thistleknot]

They have a 200m model on the repo. Maybe I'm mistaken and there is nothing that needs to be done. Wasn't sure if it's integrated in the eco system but I'll double back and check

On Thu, Jun 15, 2023, 11:55 AM amyeroberts @.***> wrote:

Hi @thistleknot https://github.com/thistleknot, thanks for opening this feature request.

Just skimming the repo, my understanding is that SpikeGPT already has a set of pretrained weights available.

If you (or someone else) would like to make this model available through the transformers API, the easiest and fastest way is to add it directly on the hub - here's a guide: https://huggingface.co/docs/transformers/custom_models.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24304#issuecomment-1593571452, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHKKOQDGGW32HFT2WGA2A3XLNLBNANCNFSM6AAAAAAZH3LM3A . You are receiving this because you were mentioned.Message ID: @.***>

[amyeroberts]

The some weights have already been uploaded on to the hub:

• https://huggingface.co/ridger/SpikeGPT-OpenWebText-216M
• https://huggingface.co/ridger/SpikeGPT-BookCorpus

However, to be able to use them with the transformers API e.g. AutoModel.from_pretrained(checkpoint), then a modeling file would also need to be created and added to the hub e.g. like this one for falcon.

[chiral-carbon]

Hi! If there is no API yet for this model may I work on it? If yes, is there a timeline for how soon one has to ship it, making it available through transformers API?

[thistleknot]

This model is available online without need for an api

On Mon, Jul 24, 2023, 12:18 PM Abhipsha Das @.***> wrote:

Hi! If there is no API yet for this model may I work on it? If yes, is there a timeline for how soon one has to ship it, making it available through transformers API?

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/24304#issuecomment-1648477014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHKKOXNLBHMUCP2GQTUK3DXR3DA7ANCNFSM6AAAAAAZH3LM3A . You are receiving this because you were mentioned.Message ID: @.***>