search

mpezeshki / pytorch_forward_forward

Implementation of Hinton's forward-forward (FF) algorithm - an alternative to back-propagation
MIT License
1.45k stars 139 forks source link

Question? How can this be applied to a basic GPT transformer text generator? #11

Open creativeautomaton opened 1 year ago

creativeautomaton commented 1 year ago

do you know of how to apply this to a GPT like model, I assumed a from scratch pytorch model.

Allaye commented 8 months ago

ff is very limiting right now, but i think it is something one can work on.

creativeautomaton commented 8 months ago

It would be very beneficial for more fine tuning be especially for edge device models like those in the STM32 model zoo ie: tensorflow lite and onnx models for micro controllers.

Last year I was interested in this and wrote to ghadia@student.matnat.uio.no a post grad who did a few sample repos on using Forward Forward from Geoffery Hintons paper examples, the MINST working code he had but Hadia's work was a bit to complex for me i think and could not get a functioning example using Forward Forward in GPT type NN.

But this would be a great subject of research and could allow for super low resource ML and open up Analog computation for not just inference of models but maybe even training.

I see it as a kind of digital/analog biology way to train machines. meh i digress.

Allaye commented 8 months ago

Agreed, the potential of FF is immense, spanning from edge devices to controllers. Currently, my team and I are researching FF, focusing on computer vision. Could you please direct me to Hadia's work? so that i could take a look.

creativeautomaton commented 8 months ago

yes sorry, here is the repo I experimented with. https://github.com/ghadialhajj/FF_unsupervised