binarize_data.py - Githubissues

QwenLM / Qwen2.5-Coder

Qwen2.5-Coder is the code version of Qwen2.5, the large language model series developed by Qwen team, Alibaba Cloud.

2.94k stars 192 forks source link

binarize_data.py #133

Closed Innovatorcloudy closed 1 month ago

Innovatorcloudy commented 1 month ago

Moved the tokenizer special token setup to a separate function Adjusted error handling for file reading Improved clarity in the argparse defaults and descriptions.