Closed quinn-dougherty closed 1 year ago
@quinn-dougherty Who is working on this project? Where is the code and what are the goals?
I definitely agree that Geva et al. is in need of a replication--also Dai et al., which builds on Geva et al. but seems even more sketchy. See my conversation with Paul Christiano here for some thoughts on things that seem sketchy about Dai et al. (and also sort of by extension Geva et al.) that would be good to try and address in a replication.
Hi, I’ll have a look into the publication and the accompanying repo (https://github.com/mega002/ff-layers) to setup GPT-NeoX for it. :-)
@MicPie Did anything ever happen with this?
Sorry, I was running out of time and I'm not sure if this project is already taken over by somebody else (there was a discussion some weeks ago).
Sorry, I was running out of time and I'm not sure if this project is already taken over by somebody else (there was a discussion some weeks ago).
Not as far as I know. Someone took a look at it but nothing came of it.
I am starting on this.
Background
What to Replicate?
The paper
Modifications
Do it with large autoregressive models
Related Papers/Frameworks
Paper: https://arxiv.org/abs/2012.14913
EleutherAI's language modeling framework: https://github.com/EleutherAI/gpt-neox
Code from the paper: https://github.com/mega002/ff-layers/