Closed shashikg closed 2 years ago
Hey,
There was a minor bug in the conversion of ## type bpe format to type. Basically, that converts the '\<unk>' to '\<unk>', which should not be done.
Colab notebook demonstrating the issue: https://colab.research.google.com/drive/1Go2_7_ugRx5g0QuyJ4EeCOCdKBtjRLwk?usp=sharing
Colab notebook demonstrating the fix: https://colab.research.google.com/drive/1oOi6juTD_kBM9Z3h9Uw1C0tcQC_Ga5K5?usp=sharing
Hey,
There was a minor bug in the conversion of ## type bpe format to type. Basically, that converts the '\<unk>' to '\<unk>', which should not be done.
Colab notebook demonstrating the issue: https://colab.research.google.com/drive/1Go2_7_ugRx5g0QuyJ4EeCOCdKBtjRLwk?usp=sharing
Colab notebook demonstrating the fix: https://colab.research.google.com/drive/1oOi6juTD_kBM9Z3h9Uw1C0tcQC_Ga5K5?usp=sharing