Closed ajesujoba closed 1 year ago
Unfortunately, the GPTNeoX model classes currently don't support adapters yet. We currently have support for GPT-2 and GPT-J. (For a full, refer to the table here: https://docs.adapterhub.ml/model_overview.html).
Thanks @calpt ! I have implemented adapter for GPTNeoX following the instructions in the documentation. It passed all tests but during the training of the language adapter, it trained the prediction head too. Do you by chance have an idea why this is happening? PR?
I was trying to train a language adapter for a
gpt-neox
I trained using the following script:then I got the following error
Your code shows that
gpt-neox
models are supported, am I doing something wrong?