Test accuracy of 0 with high loss on test set after Finetuning Titanet-large model for Speaker Identification

[Sreeni1204]

Hello Team.

I have following three issues:

1. I tried to finetune the titanet-large model using the German dataset with 274 speakers and the total data size is small. THe dataset is around 30h - 40h in total. I tried with few different configurations like - adamw optimizer with 0.0001 learning rate and angularsoftmax loss, adamw optimizer with 0.0001 learning rate and crossentropy loss, novograd optimizer with 0.0001 learning rate and crossentropy loss, sgd optimizer with 0.0001 learning rate and crossentropy loss, adamw optimizer with 0.008 learning rate and crossentropy loss, and so on. Issue: I have a small test set with 14 speakers who are common with training set, the total test size is 14minutes. The test accuracy is 0 with high loss (around 30 - 60). Please do provide some inputs.

2. I had earlier tried to finetune the TitaNet-large model with VoxCeleb German subset data. The test accuracy was around 96%. I updated the nemo-toolkit with newer version or from the main branch like two days ago and used the model from earlier to finetune the model with another German dataset with 274 speakers. I am having the following issue: ''' Error executing job with overrides: [] Error in call to target 'nemo.collections.common.losses.cross_entropy.CrossEntropyLoss': TypeError("init() got an unexpected keyword argument 'scale'") full_key: loss
   '''

3. After updating the nemo-toolkit to a newer version, I am not able to load the models that were finetuned using the earlier versions of the nemo-toolkit. Receiving the following erroe message. please provide inputs: ''' RuntimeError: Error(s) in loading state_dict for EncDecSpeakerLabelModel: Unexpected key(s) in state_dict: "decoder.final.bias". '''

Regards, Sreeni...

[nithinraok]

I have a small test set with 14 speakers who are common with training set, the total test size is 14minutes. The test accuracy is 0 with high loss (around 30 - 60). Please do provide some inputs.

Since data is very small it is diffcult to control overfitting with adam. SGD with high weight decay would be a good option. Also try increasing margin at angular loss.

Error executing job with overrides: [] Error in call to target 'nemo.collections.common.losses.cross_entropy.CrossEntropyLoss': TypeError("init() got an unexpected keyword argument 'scale'") full_key: loss

CrossEntroyLoss doesnt have scale and margin, so please remove those keys if using CE loss

Unexpected key(s) in state_dict: "decoder.final.bias".

This is because you might have trained a model with angular loss but trying to restore with model that was instantiated with CE loss

[Sreeni1204]

Thank you for the response and it helps.

I have another issue now: While testing the model, I am getting around 95% of test accuracy while doing bulk testing with all the test data using a manifest file. But when I tried to get label for audio samples one by one using the get_label() function, the model returns a single label for all the samples and do not change even if the audio sample is changed.

Could you let me know how can I debug this issuse? I am creating a webservice using the model and I need to generate the labels for each audio sample.

[nithinraok]

can you check logits with infer_file() and see if logits change?

[Sreeni1204]

Logits remain same for both get_label() and infer_file()

[nithinraok]

That shouldn't be the case, can you share the model and audio samples if possible to debug?

[Sreeni1204]

get_label() function receives the logits from the infer_file() function, shouldnt it be same in that case?

I am sorry I will get back to you on sharing the model and audio samples by end of tomorrow?

[nithinraok]

I meant logits shouldn;t be exactly same for different audio files using infer_file

_, logits1 = speaker_model.infer_file('audio1.wav')
_, logits2 = speaker_model.infer_file('audio2.wav')
assert logits1 != logits2

[Sreeni1204]

Sorry, my bad. The logits are different for different audio files. But one particular value for audio files is same, and with a low confidence.

'''label_id = logits.argmax(axis=1)'''

The label id for different audio files is same and the score of that particular label id is same.

And just to be sure, I am using the below set of code lines:

   device = self.Get_devices()
    trainer = pl.Trainer(**self.model_config.trainer, limit_train_batches=0, limit_val_batches=0)
    speaker_model = EncDecSpeakerLabelModel(cfg=self.model_config.model, trainer=trainer)
    print(speaker_model.get_label(self.wav_file))

[Sreeni1204]

Hello @nithinraok

Please find the link to the model and a few samples of audio wav files here: https://drive.google.com/drive/folders/1Axw-UeZ05PLTIbwde3FdxXCko_CAl-mZ?usp=sharing

Do request for the access to the link

[Sreeni1204]

Could you support me on this issue please?

[nithinraok]

I requested access to the drive, but haven't received it.

[Sreeni1204]

I have given the access as editor

[nithinraok]

I don;t see any issue here: (if filename corresponds to speaker labels name)

[Sreeni1204]

I am using the following code:

device = self.Get_devices() trainer = pl.Trainer(**self.model_config.trainer, limit_train_batches=0, limit_val_batches=0) speaker_model = EncDecSpeakerLabelModel(cfg=self.model_config.model, trainer=trainer) print(speaker_model.get_label(self.wav_file))

And it's not working. I get the same label for every audio file i test on.

[nithinraok]

That will create a new speaker model object.

speaker_model = EncDecSpeakerLabelModel(cfg=self.model_config.model, trainer=trainer)

This way you will get model weights from a trained model:

speaker_model = EncDecSpeakerLabelModel.restore_from(<model_path>)

[Sreeni1204]

I am using the "init_from_nemo_model" in my model configuration as given below:

init_from_nemo_model: model: path: 'Titanet_ASR_Bundestag_FT.nemo'

Shouldn't this initialize the model for testing from the nemo model that I provide?

[nithinraok]

Can you try setting like this:

init_from_nemo_model: "Titanet_ASR_Bundestag_FT.nemo"

[Sreeni1204]

Updating the model path in the the config file as suggested by you makes the model behaves the same way. The model still gives the same label for different input audio files.

[nithinraok]

I'll look at what could be the issue,but for inference that is not the suggested way. For inference, use from_pretrained, restore_from or load_from_checkpoint based on the format of the model.

[nithinraok]

Can you share full config you have used to restore the model

[Sreeni1204]

Here is the config file attached.

''' name: &name "TitaNet-Finetune" sample_rate: &sample_rate 16000

  init_from_nemo_model: "models/Titanet_ASR_Bundestag_FT.nemo"
  #  model:
  #    path: '.'

  model:
    train_ds:
      manifest_filepath: null
      sample_rate: 16000
      labels: null
      batch_size: 64
      shuffle: True
      is_tarred: False
      tarred_audio_filepaths: null
      tarred_shard_strategy: "scatter"
      augmentor:
        speed:
          prob: 0.3
          sr: *sample_rate
          resample_type: 'kaiser_fast'
          min_speed_rate: 0.95
          max_speed_rate: 1.05

    validation_ds:
      manifest_filepath: null
      sample_rate: 16000
      labels: null
      batch_size: 128
      shuffle: False

    test_ds:
      manifest_filepath: null
      sample_rate: 16000
      labels: null
      batch_size: 1
      shuffle: False
      embedding_dir: '.'
      augmentor:
        speed:
          prob: 0.3
          sr: *sample_rate
          resample_type: 'kaiser_fast'
          min_speed_rate: 0.95
          max_speed_rate: 1.05

    model_defaults:
      filters: 1024
      repeat: 3
      dropout: 0.1
      separable: true
      se: true
      se_context_size: -1
      kernel_size_factor: 1.0

    preprocessor:
      _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
      normalize: "per_feature"
      window_size: 0.025
      sample_rate: *sample_rate
      window_stride: 0.01
      window: "hann"
      features: &n_mels 80
      n_fft: 512
      frame_splicing: 1
      dither: 0.00001

    encoder:
      _target_: nemo.collections.asr.modules.ConvASREncoder
      feat_in: *n_mels
      activation: relu
      conv_mask: true

      jasper:
        -   filters: ${model.model_defaults.filters}
            repeat: 1
            kernel: [3]
            stride: [1]
            dilation: [1]
            dropout: 0.0
            residual: false
            separable: ${model.model_defaults.separable}
            se: ${model.model_defaults.se}
            se_context_size: ${model.model_defaults.se_context_size}

        -   filters: ${model.model_defaults.filters}
            repeat:  ${model.model_defaults.repeat}
            kernel: [7]
            stride: [1]
            dilation: [1]
            dropout: ${model.model_defaults.dropout}
            residual: true
            separable: ${model.model_defaults.separable}
            se: ${model.model_defaults.se}
            se_context_size: ${model.model_defaults.se_context_size}

        -   filters: ${model.model_defaults.filters}
            repeat: ${model.model_defaults.repeat}
            kernel: [11]
            stride: [1]
            dilation: [1]
            dropout: ${model.model_defaults.dropout}
            residual: true
            separable: ${model.model_defaults.separable}
            se: ${model.model_defaults.se}
            se_context_size: ${model.model_defaults.se_context_size}

        -   filters: ${model.model_defaults.filters}
            repeat: ${model.model_defaults.repeat}
            kernel: [15]
            stride: [1]
            dilation: [1]
            dropout: ${model.model_defaults.dropout}
            residual: true
            separable: ${model.model_defaults.separable}
            se: ${model.model_defaults.se}
            se_context_size: ${model.model_defaults.se_context_size}

        -   filters: &enc_feat_out 3072
            repeat: 1
            kernel: [1]
            stride: [1]
            dilation: [1]
            dropout: 0.0
            residual: false
            separable: ${model.model_defaults.separable}
            se: ${model.model_defaults.se}
            se_context_size: ${model.model_defaults.se_context_size}

    decoder:
      _target_: nemo.collections.asr.modules.SpeakerDecoder
      feat_in: *enc_feat_out
      num_classes: 274
      pool_mode: 'attention'
      emb_sizes: 192

    loss:
      _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss # you could also use cross-entrophy loss
      scale: 30
      margin: 0.2

    optim:
      name: adamw
      lr: .0001 #(original titanet-large was trained with 0.08 lr)
      weight_decay: 0.0002

      # scheduler setup
      sched:
        name: CosineAnnealing
        warmup_ratio: 0.1
        min_lr: 0.0

  trainer:
    devices: 1 # number of gpus (original titanet-large was trained on 4 nodes with 8 gpus each)
    max_epochs: 20
    max_steps: -1 # computed at runtime if not set
    num_nodes: 1
    accelerator: cpu
    strategy: ddp
    deterministic: True
    enable_checkpointing: False
    logger: False
    log_every_n_steps: 200  # Interval of logging.
    val_check_interval: 20.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
    gradient_clip_val: 20.0

  exp_manager:
    exp_dir: '../../temp_files/'
    name: iden
    create_tensorboard_logger: True
    create_checkpoint_callback: True

'''

TO update the paths for manifest files, I am using the OmegaConf commands.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[Sreeni1204]

Hello,

Any update on this issue??

[nithinraok]

It should work, have you called .maybe_init_from_pretrained_checkpoint method after creating model from config?

device = self.Get_devices()
trainer = pl.Trainer(**self.model_config.trainer, limit_train_batches=0, limit_val_batches=0)
speaker_model = EncDecSpeakerLabelModel(cfg=self.model_config.model, trainer=trainer) # cfg should have same number of decoder classes as the pretrained checkpoint
speaker_model.maybe_init_from_pretrained_checkpoint(cfg) #make sure you provided nemo model as suggested above 
print(speaker_model.get_label(self.wav_file))

# to check if both models are same or not
model2=EncDecSpeakerLabelModel.restore_from('<model>')
print(model2.get_label(self.wav_file))
assert speaker_model.state_dict()['decoder.final.weight']==model2.state_dict()['decoder.final.weight']

[nithinraok]

If your motive is to do inference don't use loading from config, it is only suggested for training as you would need to provide training and validation manifests, which are not expected during inference.