Closed tom1997 closed 1 year ago
I have different dataset manifests, and I join them with tokenizer.py. and I wanna know is unique_text_tokens.k2symbols would affect the training? cuz the index is not the same
All your need is merging the symbols before training.
I have different dataset manifests, and I join them with tokenizer.py. and I wanna know is unique_text_tokens.k2symbols would affect the training? cuz the index is not the same