search

NVIDIA / NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs
Apache License 2.0
478 stars 57 forks source link

Translation example with ctranslate2's Translator. #246

Open uahmed93 opened 1 week ago

uahmed93 commented 1 week ago

As we have added support for HF model translation via CrossFit, we are working towards performance improvement with ctranslate2. This work depends on adding support for ctranslate2 in CrossFit, and then will need to create pipeline for this work in NDC.(Draft PR)

With a workaround for ctranslate2 in CrossFit, huge performance improvement was seen. On single GPU, following is the performance : Experiment Standalone pytorch inference Standalone + ctranslate2 Crossfit+ctranslate2
Inference time ~1hr 50mins 23min 54sec 6min 29sec (including extra processing for workarund : 3sec)
BLEU score - 0.9585 0.9586

BLEU score was calculated w.r.t Standalone pytorch inference on 74058 sentences.

VibhuJawa commented 1 week ago

CC: @arhamm1 for awareness for the work here

uahmed93 commented 4 days ago

Added an example notebook here