👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
w5688414
commented
7 months ago
请提出你的问题
https://huggingface.co/facebook/opt-13b/blob/main/tokenizer_config.json 从这个链接获取了OPT-13b的模型参数以及各种配置后调用
python -u -m paddle.distributed.launch --gpus "0,1" finetune_generation.py ./opt/lora_argument.json进行训练，但是报了不支持GPT2Tokenizer的错误，我检查了PaddleNLP/paddlenlp/transformers/auto/tokenizer.py这个文件，也没有发现opt模型对应的tokenizer，请问我应该用哪个tokernizer进行训练呢？