AttributeError: 'GPTNeoXConfig' object has no attribute 'adapters'

[ajesujoba]

I was trying to train a language adapter for a gpt-neox I trained using the following script:

OUTDIR="."
python run_clm.py \
        --language de \
        --model_name_or_path gpt-neox/hf_model/gpt125M \
        --train_file train.txt \
        --validation_file  dev.txt \
        --output_dir $OUTDIR/de \
        --do_train \
        --do_eval \
        --per_device_train_batch_size 4 \
        --per_device_eval_batch_size 8 \
        --gradient_accumulation_steps 1 \
        --learning_rate 5e-5 \
        --max_steps 100000 \
        --num_train_epochs 100 \
        --save_steps 10000000 \
        --overwrite_output_dir \
        --train_adapter \
        --adapter_config pfeiffer+inv \
        --adapter_reduction_factor 2 \
        --evaluation_strategy steps \
        --eval_steps 1000000 \
        --load_best_model_at_end \
        --save_total_limit 1

then I got the following error

Traceback (most recent call last):                          
  File "run_clm.py", line 633, in <module>                                                                                                                                                                                                                                         
    main()                                                                                                                                                                                                                                                                         
  File "run_clm.py", line 398, in main                                                                                                                                                                                                                                             
    if task_name not in model.config.adapters:                                                                                                                                                                                                                                     
  File "/nethome/jalabi/anaconda3/envs/madx/lib/python3.8/site-packages/transformers/configuration_utils.py", line 254, in __getattribute__                                                                                                                                        
    return super().__getattribute__(key)                                                                                                 
AttributeError: 'GPTNeoXConfig' object has no attribute 'adapters'

Your code shows that gpt-neox models are supported, am I doing something wrong?

[calpt]

Unfortunately, the GPTNeoX model classes currently don't support adapters yet. We currently have support for GPT-2 and GPT-J. (For a full, refer to the table here: https://docs.adapterhub.ml/model_overview.html).

[ajesujoba]

Thanks @calpt ! I have implemented adapter for GPTNeoX following the instructions in the documentation. It passed all tests but during the training of the language adapter, it trained the prediction head too. Do you by chance have an idea why this is happening? PR?