Open sankar-mukherjee opened 1 month ago
Have you tried loading the 1.1b using
from nemo.collections.asr import ASRModel
model = ASRModel.from_pretrained('nvidia/parakeet-tdt-1.1b')
And see the memory usage? You would need twice this size initially as you are looking to finetune from existing model. Memory usage at this point probably would answer if you have anymore memory left to train.
I am trying to finetune nvidia/parakeet-tdt-1.1b model using the instruction below with g5.12xlarge with 4 gpus with 24gb memory.
https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/configs.html#fine-tuning-configurations
First I have created a docker container and inside the container i am running
finetune.sh
. I am getting OutOfMemoryError before starting the traning. I have tried reducing the batch_size = 16, 8, 4, 2, as well as max_duration of the audio files to = 20, 10 , 5 . None of them succeeds. Can anyone help me?Docker file
requirement.txt
finetune.sh
Error: