search

jia-zhuang / chinese-llama2.c

支持中文的 llama2.c
MIT License
8 stars 1 forks source link

Add Hugging Face Export Support #5

Closed fangzhangmnm closed 1 month ago

fangzhangmnm commented 1 month ago

把训练的模型导出到HuggingFace

python3 export.py ./hf_export/my_model --version -1 --dtype fp16 --checkpoint ./out/ckpt.pt
python3 export_tokenizer.py ./hf_export/my_model -m ./data/tok5000.model

注意：使用TinyStories.py生成的数据集训练出来的模型可能会在写完一个故事之后输出乱码。这是因为训练数据中只有bos token没有eos token的缘故。

fangzhangmnm commented 1 month ago

Oh I'm sorry I forgot to clean up some code in the first commit. can you ignore them?

jia-zhuang commented 1 month ago

I've already merged the request. I'd like to invite you to be administrator of the project, feel free to update the code.

------------------ 原始邮件 ------------------ 发件人: "jia-zhuang/chinese-llama2.c" @.>; 发送时间: 2024年6月5日(星期三) 中午11:16 @.>; @.***>; 主题: Re: [jia-zhuang/chinese-llama2.c] Add Hugging Face Export Support (PR #5)

Oh I'm sorry I forgot to clean up some code in the first commit. can you ignore them?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>