Improve CTranslate2 wrapping in translation_server

OpenNMT / OpenNMT-py

Open Source Neural Machine Translation and (Large) Language Models in PyTorch

https://opennmt.net/

MIT License

6.7k stars 2.25k forks source link

Improve CTranslate2 wrapping in translation_server #2001

Open francoishernandez opened 3 years ago

francoishernandez commented 3 years ago

https://forum.opennmt.net/t/ctranslate2-on-opennmt-py-server/4175/8

guillaumekln commented 3 years ago

After reviewing the code, here's what could be improved:

[x] Make the following translator parameters configurable:
- inter_threads
- intra_threads
- compute_type
[ ] Allow parallel translations as supported by CTranslate2: I tried to enable that but even though the waitress module is multi-threaded and accepts concurrent requests, it seems the requests are then processed sequentially
[ ] Revise the unloading mechanism to not assume the model is running on the GPU
[ ] Maybe cleanup the initial dummy translation: the first translation has a higher latency on GPU but this was improved in recent versions (I think it's around 200 ms now)

guillaumekln commented 2 years ago

I tried to enable that but even though the waitress module is multi-threaded and accepts concurrent requests, it seems the requests are then processed sequentially

I did not realize that the translation method is inside a critical section. Note this is not needed for CTranslate2: the translation and model loading/unloading are fully thread safe. So removing the critical section for CTranslate2 can improve the scalability of the server for CPU translations with inter_threads > 1 and multi-GPU translations.

vince62s commented 1 year ago

@francoishernandez @pltrdy do you recall why this https://github.com/OpenNMT/OpenNMT-py/pull/1108 was introduced ? threads memoery leakages ?

pltrdy commented 1 year ago

I think that in the translation server loading/unloading and even running a model was not thread safe. I don't know anything about CTranslate 2 tho, so I can't tell how they differ

souleymanefall176 commented 9 months ago

"Good evening. I have an issue. When I run the command (ct2-opennmt-py-converter --model_path averaged-10-epoch.pt --output_dir ende_ctranslate2 --quantization int8), I get this error (ModuleNotFoundError: No module named 'onmt.inputters.text_dataset')."