missing `model_type` key in config.json

[xiyanghu]

When I tried to run the training, it raises an error for missing model_type key in config.json.

The training script I used is the one in the run-ccp.sh:

outdir=runs/CCP
model_cfg=data/ccp

python -m torch.distributed.launch --nproc_per_node=8 train_dense_encoder.py \
--max_grad_norm 2.0 --encoder_model_type hf_bert --pretrained_model_cfg ${model_cfg} \
--seed 42 --sequence_length 256 --warmup_steps 1237 --batch_size 6 --dev_batch_size 32 \
--train_file data/nq/biencoder-nq-train.json --dev_file data/nq/biencoder-nq-dev.json \
--output_dir ${outdir} --learning_rate 1e-05 --num_train_epochs 20 --fp16 --pooling cls

This is the error message:

[2022-08-05 01:46:01] [6 - ERROR] [train_dense_encoder.py - 667] Unrecognized model in data/ccp. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: visual_bert, roformer, clip, bigbird_pegasus, deit, luke, detr, gpt_neo, big_bird, speech_to_text, vit, wav2vec2, m2m_100, convbert, led, blenderbot-small, retribert, ibert, mt5, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, megatron_bert, mpnet, bart, blenderbot, reformer, longformer, roberta, deberta-v2, deberta, flaubert, fsmt, squeezebert, hubert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag, tapas
Traceback (most recent call last):
  File "train_dense_encoder.py", line 661, in main
    trainer = BiEncoderTrainer(args)
  File "train_dense_encoder.py", line 85, in __init__
    tensorizer, model, optimizer = init_biencoder_components(args.encoder_model_type, args)
  File "/home/ec2-user/efs/users/xiyanghu/CCP_IJCAI22/dpr/models/__init__.py", line 46, in init_biencoder_components
    return init_comp(BIENCODER_INITIALIZERS, encoder_type, args, **kwargs)
  File "/home/ec2-user/efs/users/xiyanghu/CCP_IJCAI22/dpr/models/__init__.py", line 40, in init_comp
    return initializers_dict[type](args, **kwargs)
  File "/home/ec2-user/efs/users/xiyanghu/CCP_IJCAI22/dpr/models/__init__.py", line 19, in init_hf_bert_biencoder
    return get_bert_biencoder_components(args, **kwargs)
  File "/home/ec2-user/efs/users/xiyanghu/CCP_IJCAI22/dpr/models/hf_models.py", line 28, in get_bert_biencoder_components
    question_encoder = HFBertEncoder(args)
  File "/home/ec2-user/efs/users/xiyanghu/CCP_IJCAI22/dpr/models/hf_models.py", line 72, in __init__
    cfg = AutoConfig.from_pretrained(args.pretrained_model_cfg)
  File "/home/ec2-user/efs/users/xiyanghu/envs_ccp/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 457, in from_pretrained
    f"Unrecognized model in {pretrained_model_name_or_path}. "
ValueError: Unrecognized model in data/ccp. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: visual_bert, roformer, clip, bigbird_pegasus, deit, luke, detr, gpt_neo, big_bird, speech_to_text, vit, wav2vec2, m2m_100, convbert, led, blenderbot-small, retribert, ibert, mt5, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, megatron_bert, mpnet, bart, blenderbot, reformer, longformer, roberta, deberta-v2, deberta, flaubert, fsmt, squeezebert, hubert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag, tapas

[xiyanghu]

Another issue:

It seems like the google drive folder does not contain the tokenizer.json file?

[xiyanghu]

In addition to the pre-trained model checkpoint, could you also provided a checkpoint of trained retrieval? Thank you!

[wuning0929]

Sorry for delay, I have updated config.json, and tokenizer.json. Actually it's the same to the XLMR-Large in Hugginface.