Add the helper function in Keras setup to clean up special tokens in GTP2 and SentencePiece tokenizer.

PAIR-code / lit

The Learning Interpretability Tool: Interactively analyze ML models to understand their behavior in an extensible and framework agnostic interface.

https://pair-code.github.io/lit

Apache License 2.0

3.45k stars 351 forks source link

Closed copybara-service[bot] closed 3 months ago

copybara-service[bot] commented 3 months ago

Add the helper function in Keras setup to clean up special tokens in GTP2 and SentencePiece tokenizer.