Closed LL-AI-dev closed 4 months ago
After some fiddling, I was able to get the nemo2riva part of the deployment working using a riva docker image. It can create an environment that allows the conversion of .nemo to .riva files for both pretrained and finetuned (regular and adapter variants) of FastPitch and HifiGan.
The step to create a .rmir file is also successful in that the riva-build command completes without error. However when this model is being deployed by bash riva_start.sh, we get the new error:
failed to load 'riva-onnx-fastpitch_encoder-Jaz_v1' version 1: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-Jaz_v1/1/model.onnx failed:/workspace/onnxruntime/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) Unsupported model IR version: 9, max supported IR version: 8
(The full docker log is listed at the bottom)
DOCKERFILE
FROM nvcr.io/nvidia/riva/riva-speech:2.14.0-servicemaker
#make sure pip ist installed
RUN apt update && apt install python3-pip -y
#get a simple fastpitch model to convert
RUN wget --content-disposition \
'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/nemo/tts_en_fastpitch/IPA_1.13.0/files?redirect=true&path=tts_en_fastpitch_align_ipa.nemo' \
-O tts_en_fastpitch_align_ipa.nemo
#install NeMo dependencies
RUN apt-get update && apt-get install -y libsndfile1 ffmpeg
RUN pip install Cython
#install nemo2riva dependencies
RUN pip install nvidia-pyindex
#install NeMo
RUN git clone https://github.com/NVIDIA/NeMo
WORKDIR NeMo
RUN git switch 'r1.23.0'
RUN pip install -e .
WORKDIR ..
#install some other required packages
RUN pip install matplotlib
RUN pip install einops
RUN pip install transformers
RUN pip install pandas
RUN pip install inflect
RUN pip install typing_extensions==4.7.1
RUN pip install wandb
RUN pip install youtokentome
RUN pip install editdistance
RUN pip install nemo_text_processing
RUN pip install lhotse
RUN pip install pyannote.audio
RUN pip install webdataset
RUN pip install datasets
RUN pip install jiwer
#install nemo2riva
RUN pip install nemo2riva==2.14.0
#fix the errors arising due to nemo requiring python 3.10
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/common/tokenizers/canary_tokenizer.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/asr/data/audio_to_text_lhotse.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/asr/data/audio_to_text_lhotse_prompted.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/common/data/lhotse/nemo_adapters.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/common/data/lhotse/dataloader.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/asr/models/aed_multitask_models.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/asr/data/huggingface/hf_audio_to_text.py
RUN sed -i '1i from __future__ import annotations' /usr/local/lib/python3.8/dist-packages/nemo/collections/asr/parts/submodules/rnnt_greedy_decoding.py
@LL-AI-dev the failure on deploy is very similar to what I saw deploying ASR models. It turned out to be caused by a discrepancy in the version of onnx that NeMo uses and the one that is used by Riva/Triton (older); see this issue https://github.com/nvidia-riva/nemo2riva/issues/36. For ASR the workaround was to either downgrade the onnx library in NeMo or avoid using the onnx runtime in Riva.
@itzsimpl this is my script for testing different versions of onnx. before the script is run, i do a simple pip install onnx==1.13.0 as you mentioned in [https://github.com/nvidia-riva/nemo2riva/issues/36] but this has seemingly not solved the issue. I also tried with different versions of onnxruntime to no effect.
import nemo
from nemo.collections.tts.models import FastPitchModel, HifiGanModel
import onnx
nemo_ver = nemo.__version__
onnx_ver = onnx.__version__
print("onnxruntime: ",onnx_ver)
print("nemo: ",nemo_ver)
fp_model_name = f"FastPitch_nemo{nemo_ver}_onnx{onnx_ver}"
hg_model_name = f"HifiGan_nemo{nemo_ver}_onnx{onnx_ver}"
model = FastPitchModel.from_pretrained(model_name="tts_en_fastpitch_ipa")
model.save_to(f'/models/{fp_model_name}.nemo')
model = HifiGanModel.from_pretrained(model_name="tts_en_hifigan")
model.save_to(f'/models/{hg_model_name}.nemo')
print("models saved")
import os
os.system(f"nemo2riva --out /models/riva/{fp_model_name}.riva --key tlt_encode /models/{fp_model_name}.nemo")
os.system(f"nemo2riva --out /models/riva/{hg_model_name}.riva --key tlt_encode /models/{hg_model_name}.nemo")
cmd = f"""
riva-build speech_synthesis --force \
/models/rmir/tts_pipeline_model__nemo{nemo_ver}_onnx{onnx_ver}.rmir:tlt_encode \
/models/riva/{fp_model_name}.riva:tlt_encode \
/models/riva/{hg_model_name}.riva:tlt_encode \
--language_code=en-US \
--num_speakers=1 \
--voice_name test_{onnx_ver} \
--sample_rate 22050 \
--abbreviations_file=/riva_aux/abbr.txt \
--wfst_tokenizer_model=/riva_tn/tokenize_and_classify.far \
--wfst_verbalizer_model=/riva_tn/verbalize.far \
--phone_set=ipa \
--phone_dictionary_file=/riva_aux/ipa_cmudict-0.7b_nv22.08.txt \
--upper_case_chars=True
"""
os.system(cmd)
My next thought is to test using trt_32 or torchscript via the args in the riva-build command for the encoder (as thats where the error seems to be) or for both enc and dec if needed. [https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-custom.html#riva-build-optional-parameters]
using the flag "use_trt_fp32" during riva-build results in a model that when deployed gets the same error as before about the IR version.
UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-Jaz_v1/1/model.onnx failed:/workspace/onnxruntime/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) Unsupported model IR version: 9, max supported IR version: 8
When using the flag "use_torchscript" there is an issue during riva_init.sh due to a missing file. This is not resolved by using --FORMAT TS
during the nemo2riva call. infact, setting the format to TS results in the main folder in the output location being empty.
Despite the lack of success before I will go back to trying a combination of downgraded onnx and onnxruntime libraries. As I do not seem to be able to avoid the use of onnx as @itzsimpl suggested.
all with nemo 1.23.0. versions of onnx/onnx runtime were set using pip install X commands in the docker file, before the script above is run to download models & save them (I believe saving them in this way is enough to update the model IR version used by onnxruntime. between onnx 1.13.0 and 1.14.0, and onnxruntime 1.16.3, 1.15.1, 1.14.0, 1.13.1 I was not able to find a combination that would allow the creation of a .rmir file that could be successfully deployed.
Tomorrow I will try the following approaches to resolve this:
onnx 1.14.0, onnxruntime 1.16.3 (defaults from dockerfile)
onnx 1.13.0, onnxruntime 1.16.3
onnx 1.13.0, onnxruntime 1.15.1 -UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test__onnx1.13.0_onnxrt1.15.1/1/model.onnx failed:Invalid tensor data type 0. (accidentally ran twice, same result)
onnx 1.14.0, onnxruntime 1.15.1 -UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test__onnx1.14.0_onnxrt1.15.1/1/model.onnx failed:/workspace/onnxruntime/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) Unsupported model IR version: 9, max supported IR version: 8
onnx 1.14.0, onnxruntime 1.14.0 -UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test__onnx1.14.0_onnxrt1.14.0/1/model.onnx failed:/workspace/onnxruntime/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) Unsupported model IR version: 9, max supported IR version: 8
onnx 1.14.0, onnxruntime 1.13.1 -UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test__onnx1.14.0_onnxrt1.13.1/1/model.onnx failed:/workspace/onnxruntime/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) Unsupported model IR version: 9, max supported IR version: 8
Using pre-trained models DIRECTLY from ngc is ok: onnx 1.13.0 onnxruntime 1.15.1 nemo_toolkit 1.20.0 (pip install nemo_toolkit[all]==1.20.0)
wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/nemo/tts_hifigan/1.0.0rc1/files?redirect=true&path=tts_hifigan.nemo' -O tts_hifigan.nemo
wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/nemo/tts_en_fastpitch/IPA_1.13.0/files?redirect=true&path=tts_en_fastpitch_align_ipa.nemo' -O tts_en_fastpitch_align_ipa.nemo
nemo2riva --out ../nemo_models/riva/tts_en_fastpitch_align_ipa.riva --key tlt_encode tts_en_fastpitch_align_ipa.nemo
nemo2riva --out ../nemo_models/riva/tts_hifigan.riva --key tlt_encode tts_hifigan.nemo
#below is run within the docker container after it is launched
riva-build speech_synthesis --force \
/models/rmir/tts_pipeline_simple.rmir:tlt_encode \
/models/riva/tts_en_fastpitch_align_ipa.riva:tlt_encode \
/models/riva/tts_hifigan.riva:tlt_encode \
--language_code=en-US \
--num_speakers=1 \
--voice_name simple \
--sample_rate 22050 \
--abbreviations_file=/riva_aux/abbr.txt \
--wfst_tokenizer_model=/riva_tn/tokenize_and_classify.far \
--wfst_verbalizer_model=/riva_tn/verbalize.far \
--phone_set=ipa \
--phone_dictionary_file=/riva_aux/ipa_cmudict-0.7b_nv22.08.txt \
--upper_case_chars=True
When I ran this combo previously with a higher nemo version, 1.23 (installed via git clone + pip install -e) and saved the models through nemo before doing the conversion, I was getting some error related to the tensor data type. This implies the data type error is resulting from the interaction with nemo.
Rerunning the above code:
Will be doing some testing but it seems that a working combination to deploy nemo models requires:
when i have narrowed down the specific requirements I will post a dockerfile that can be used for training and converting.
Overall, it would be great if the nemo, nemo2riva, and riva packages can be made more compatible
After testing many combinations of packages, I have discovered the current compatibility for training and converting .nemo files is limited to:
Training Environment:
.nemo
file.Converting Environment:
Until the conflicts are resolved, it will not possible for models or functions introduced in nemo after version 1.20 to be converted to a .riva file
@LL-AI-dev could you please try "pip install nvidia-eff>=0.6.4 nvidia-eff-tao-encryption>=0.1.8" in your python3.10 environment and see whether that helps with this error? Thanks.
@rmittal-github thanks for the suggestion. that does allow the nemo2riva (version 2.14) conversion to run, but when deploying the downstream .rmir file, it gets the error:
UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test_py310_onnx13/1/model.onnx failed:Invalid tensor data type 0.
I suspect this might be related to a torch version issue (from riva release logs, 2.14 known issues)
When generating .riva models from .nemo using nemo2riva, the nemo:23.08 image is not compatible with Riva due to updated torch version. To avoid any Riva deployment issues, the recommendation is to continue using the last working nemo image
these are the pytorch versions in question:
Do you have any suggestions to resolve this?
I tried downgrading nemo2riva to version 2.13 as that had resolved the data type error previously
- Package version for nemo2riva: 2.13.0 is required.
- Any higher fails with the error related to “Invalid tensor data type 0”. Even 2.13.1 fails.
downgrading results in the error:
Traceback (most recent call last):
File "/usr/local/bin/nemo2riva", line 8, in <module>
sys.exit(nemo2riva())
File "/usr/local/lib/python3.10/dist-packages/nemo2riva/cli/nemo2riva.py", line 49, in nemo2riva
Nemo2Riva(args)
File "/usr/local/lib/python3.10/dist-packages/nemo2riva/convert.py", line 42, in Nemo2Riva
trainer = Trainer(cfg_trainer)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults
return fn(self, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 399, in __init__
self._accelerator_connector = _AcceleratorConnector(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 140, in __init__
self._check_config_and_set_final_flags(
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 222, in _check_config_and_set_final_flags
raise ValueError(
ValueError: You selected an invalid accelerator name: `accelerator=TrainerConfig(logger=False, callbacks=None, default_root_dir=None, gradient_clip_val=0, num_nodes=1, enable_progress_bar=True, overfit_batches=0.0, check_val_every_n_epoch=1, fast_dev_run=False, accumulate_grad_batches=1, max_epochs=1000, min_epochs=1, max_steps=-1, min_steps=None, limit_train_batches=1.0, limit_val_batches=1.0, limit_test_batches=1.0, val_check_interval=1.0, log_every_n_steps=50, accelerator='cpu', sync_batchnorm=False, precision=32, num_sanity_val_steps=2, profiler=None, benchmark=False, deterministic=False, use_distributed_sampler=True, detect_anomaly=False, plugins=None, limit_predict_batches=1.0, gradient_clip_algorithm='norm', max_time=None, reload_dataloaders_every_n_epochs=0, devices='auto', strategy='auto', enable_checkpointing=False, enable_model_summary=True, inference_mode=True, barebones=False)`. Available names are: auto, cpu, cuda, hpu, ipu, mps, tpu.
i attempt to change the line where the accelerator is set in /usr/local/lib/python3.10/dist-packages/nemo2riva/convert.py
but regardless of which value I tried, the conversion failed
@LL-AI-dev I think downgrading to 2.13 is not required. Could you please try with nemo:23.06 image and 2.14.0, along with the pip install packages I mentioned in my last comment, and see if that resolves the issue?
As per the 2.14.0 release notes known issues, nemo:23.08 is not compatible and need to try previous nemo image (23.06)
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue was closed because it has been inactive for 7 days since being marked as stale.
Hardware - GPU (T4) Hardware - CPU Operating System - ubuntu 20.04 running on AWS EC2 g4dn.2xlarge instance
I am currently trying to convert a model (several of different types but for now not even a FastPitch model is working) In the past i had deployed several nemo pipelines to riva but that developing environment was lost during some updates and I have not been able to convert and deploy any nemo models since. I believe this lost environment was using nemo 1.20.0 and riva & nemo2riva version 2.13.1, however using those versions does not seem to work for me anymore.
Recently I have been testing several versions of nemo, nemo2riva and riva using the dockerfile below in order to deploy models. (will update with testing data as I continue to try and retry combinations)
As you can see in the dockerfile, it is using a pretrained model from ngc, however I get the same error even on a .nemo model that was trained using the latest nemo version
The error relates to nvidia-eff and being unable to encrypt the model. This error is consistent regardless of the nemo image used. I have also tried using a pytorch base image but this results in the same errors. I tried using a riva-servicemaker image too but that had an issue arising because the image uses python 3.8 but nemo has required 3.10 for a long time.
I really need some help resolving this as development is being delayed as I cannot update any ASR, NLP, or TTS models currently. How can I resolve this?