What is the difference between wordpieces and byte pair encoding?

tensorflow / nmt

TensorFlow Neural Machine Translation Tutorial

Apache License 2.0

6.36k stars 1.96k forks source link

What is the difference between wordpieces and byte pair encoding? #384

Closed yapingzhao closed 5 years ago

yapingzhao commented 5 years ago

Hi,

I want to know the difference and connection between wordpieces and byte pair encoding's two language processing techniques. Looking forward to your advice or answers. Best regards, Thank you very much!