代码中已经有设置：
if tokenizer.pad_token_id is None:
if tokenizer.unk_token_id is not None:
tokenizer.pad_token = tokenizer.unk_token
else:
tokenizer.pad_token = tokenizer.eos_token
则呢蜜还会提示pad_token 未定义呢，issues中每找到相关解决方法
raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
ValueError: Cannot handle batch sizes > 1 if no padding token is defined.
XiaozhuLove
commented
2 months ago
代码中已经有设置： if tokenizer.pad_token_id is None: if tokenizer.unk_token_id is not None: tokenizer.pad_token = tokenizer.unk_token else: tokenizer.pad_token = tokenizer.eos_token
则呢蜜还会提示pad_token 未定义呢，issues中每找到相关解决方法
ValueError: Cannot handle batch sizes > 1 if no padding token is defined.