search

glample / fastBPE

Fast BPE
MIT License
656 stars 96 forks source link

learnbpe and the process is killed #22

Open tuanshanyou opened 5 years ago

tuanshanyou commented 5 years ago

the corpus is 4G,and the memory is 16G i guess that it's KILLED because the memory is full. how to deal with it if it do not reduce the ncodes?

Oktai15 commented 5 years ago

@tuanshanyou , you can try this new library: https://github.com/VKCOM/YouTokenToMe. This library is more efficient than fastBPE.