Open mikkokotila opened 4 years ago
What I can think of right now is first running the preprocessing on the input string (here), then distributing pieces of the generated chunks to be tokenized to different threads, and concatenating in the end the lists of Tokens of each worker.
The preprocessing is much faster than the actual tokenizing, and right now, I don't have any idea on how to split this task for concurrent threads...
Does that help?
Unfortunately we cannot achieve parallelization with multi-threading in python because og the GIL
(Global Lock Interpreter), which only allow one thread to run on a process. But we can do mutli-processing to achieve parallelization, which I have implemented in the script to do Google OCR on bunch of images.
We will be adding this multi-processing soon
A simple way of doing it would be to change this line: https://github.com/Esukhia/botok/blob/improve-tok/botok/tokenizers/wordtokenizer.py#L80.
Creating a new method that returns tokens
where all the multiprocessing happens will keep things simple, and would even allow for two modes to choose from: single processing or multi-processing.
The below example seems to work. The benefit is that it does not require any changes to Botok. The trick is to find the right number for Pool
. I could do around 20 on a 16-core machine, but after that massive memory leak takes over. It seems that Pool
has a lot of issues with memory, so that's probably expected. It seems that performance gain is linear. 16x processes, 16x faster tokenization.
I have OpenPecha repos in texts/
so that just the directory containing the volumes an the volumes are there.
from botok import TokChunks
def tokenization(text, tokenizer):
out = []
preproc = TokChunks(text)
preproc.serve_syls_to_trie()
tokens = tokenizer.tokenize(preproc)
for i in range(len(tokens)):
out.append(tokens[i]["text"])
del preproc, tokens
return out
def init_tokenizer():
from botok import BoSyl, Config, Tokenize, Trie
config = Config()
trie = Trie(BoSyl,
profile=config.profile,
main_data=config.dictionary,
custom_data=config.adjustments,
pickle_path=config.dialect_pack_path.parent)
tokenizer = Tokenize(trie)
return tokenizer
def create_paths():
import os
paths = []
for text_dir in os.listdir('texts'):
for text in os.listdir('texts/' + text_dir):
paths.append(text_dir + '/' + text)
return paths
def create_tokens(path):
import os
import gc
# handle input
docs = open('texts/' + path, 'r').read()
tokens = tokenization(docs, tokenizer)
# handle output
os.makedirs(os.path.dirname('tokens/' + path), exist_ok=True)
out = open('tokens/' + path, 'w')
out.write(' '.join(tokens))
del tokens
out.close()
gc.collect()
print(path)
from multiprocessing import Pool
import warnings
warnings.simplefilter('ignore')
tokenizer = init_tokenizer()
p = Pool(20)
paths = create_paths()
p.map(create_tokens, paths)
:)
Nice!
Currently we're running everything on a single thread. I wonder if there is a straightforward way to provide a wrapper that allows multi-threading (or even distributing) tokenization.