How to create nemo_text_processing/text_normalization/en/data/numbers/cardinal_number_name.far?

NVIDIA / NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html

Apache License 2.0

11.95k stars 2.48k forks source link

How to create nemo_text_processing/text_normalization/en/data/numbers/cardinal_number_name.far? #2870

Closed npuichigo closed 3 years ago

npuichigo commented 3 years ago

Is there anyway in to create the fst for cardinal number name using pynini like other cases? Or can anyone provide a way to create that, since I would like to do some locale expansion.

npuichigo commented 3 years ago

@yzhang123 could u help to take a look?

yzhang123 commented 3 years ago

there is a way, this is mainly by using the same code as cardinal wfst from inverse text normalization but inverting the graph at the end and fixing some format issues. But that code is very slow. So we did optimization and directly load the far. If you like the previous inline code version take a look at this PR.

https://github.com/NVIDIA/NeMo/pull/2180/files

npuichigo commented 3 years ago

Thanks for reply. You said there’s some optimization in the far for direct loading. Could you also provide some details about that?

yzhang123 commented 3 years ago

you can for example the graph.optimize() in far file and load it, so you don't have to build it all the time. We have a cache function currently too, so when you specify --cache_dir it will save the generated .far file and load that next time. that is much faster.