Open aisk opened 1 year ago
The default_bpe() is called as default parameter in constructor of SimpleTokenizer. Since python's function default parameters only execute once when the function definition is executed, so there is no need to cache it, and the operation is not so expensive.
default_bpe()
The
default_bpe()is called as default parameter in constructor of SimpleTokenizer. Since python's function default parameters only execute once when the function definition is executed, so there is no need to cache it, and the operation is not so expensive.