search

OptimalScale / LMFlow

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.
https://optimalscale.github.io/LMFlow/
Apache License 2.0
8.11k stars 819 forks source link

[Bug fix] Tokenization multiprocessing fix #845

Closed wheresmyhair closed 4 weeks ago

wheresmyhair commented 4 weeks ago

  1. Tokenization multiprocessing bug fix fix

  2. Disable passing a ConversationTemplate object directly through model.tokenize() method. For customizing conversation templates, please refer to doc.