search

shikras / shikra

Other
710 stars 44 forks source link

RuntimeError: Internal: unk is not define #29

Closed Lanxin1011 closed 11 months ago

Lanxin1011 commented 11 months ago

Dear authors,

Thanks for the great work! I've encountered a RuntimeError: Internal: unk is not define when running 


        --main_process_port 23786 \
        mllm/pipeline/finetune.py \
        config/shikra_pretrain_final19_stage2.py \
        --cfg-options model_args.model_name_or_path=/path/to/init/checkpoint```
and the **Traceback** indicates that something went wrong when calling `transformers.AutoTokenizer.from_pretrained()`.

Do you have any idea of this Error? 

Looking forward to your reply : )
Lanxin1011 commented 11 months ago

I‘ve solved this problem according to https://github.com/facebookresearch/llama/issues/287/, which is due to the damage of tokenizer.model.