Closed triumph9989 closed 1 year ago
Hi, @triumph9989 . Thank you for uploading the feedback. Please make sure you do not label this type of topics as "bug" label since the software is not showing any signs of malfunctioning features. NeMo contributors should differentiate the actual bug from some topics for debate.
Can you also share the data preparation steps? Especially the segment lengths and the size of the fine-tuning data.
Also please share your evaluation setup. default is collar=0.25, ignore_overlap=True.
@tango4j Sorry about not changing the label, and thanks for concern my issue. The followings are my checkpoints
'TitaNet-Finetune-encoder-cn2-300-adjust--val_loss=0.1640-epoch=9-last.ckpt'
'TitaNet-Finetune-encoder-cn2-300-adjust--val_loss=0.1668-epoch=6.ckpt'
'TitaNet-Finetune-encoder-cn2-300-adjust--val_loss=0.1640-epoch=9.ckpt'
TitaNet-Finetune-encoder-cn2-300-adjust.nemo
'TitaNet-Finetune-encoder-cn2-300-adjust--val_loss=0.1657-epoch=8.ckpt'
I used the TitaNet-Finetune-encoder-cn2-300-adjust.nemo
and it get the worse Confusion Error rate.
After, I used'TitaNet-Finetune-encoder-cn2-300-adjust--val_loss=0.1640-epoch=9.ckpt'
and the Confusion Error rate was better than using pre-trained titanet-large, but I don't know why.
offline_diarization_with_asr.yaml
name: &name "ClusterDiarizer"
num_workers: 4
sample_rate: 16000
batch_size: 64
diarizer:
manifest_filepath: /workspace/nemo/examples/speaker_tasks/diarization/manifest_Ali_far.json #manifest_Ali_far.json
out_dir: output_SD_ASR_VAD_TH_100_ali_far_trimmed_test
oracle_vad: False # If True, uses RTTM files provided in the manifest file to get speech activity (VAD) timestamps
collar: 0.25 # Collar value for scoring
ignore_overlap: True # Consider or ignore overlap segments while scoring
vad:
model_path: null # .nemo local model path or pretrained model name or none
external_vad_manifest: null # This option is provided to use external vad and provide its speech activity labels for speaker embeddings extraction. Only one of model_path or exx
ternal_vad_manifest should be set
parameters: # Tuned parameters for CH109 (using the 11 multi-speaker sessions as dev set)
window_length_in_sec: 0.15 # Window length in sec for VAD context input
shift_length_in_sec: 0.01 # Shift length in sec for generate frame level VAD prediction
smoothing: "median" # False or type of smoothing method (eg: median)
overlap: 0.875 # Overlap ratio for overlapped mean/median smoothing filter
onset: 0.4 # Onset threshold for detecting the beginning and end of a speech
offset: 0.7 # Offset threshold for detecting the end of a speech
pad_onset: 0.05 # Adding durations before each speech segment
pad_offset: -0.1 # Adding durations after each speech segment
min_duration_on: 0.2 # Threshold for small non_speech deletion
min_duration_off: 0.2 # Threshold for short speech segment deletion
filter_speech_first: True
speaker_embeddings:
model_path: titanet_large # .nemo local model path or pretrained model name (titanet_large, ecapa_tdnn or speakerverification_speakernet) TitaNet-Finetune-cn2-300-30ep-lr1-e5.nn
emo, /workspace/nemo/examples/speaker_tasks/recognition/exp/TitaNet-Finetune-encoder-cn2-300-adjust/2022-10-28_05-48-05/checkpoints/TitaNet-Finetune-encoder-cn2-300-adjust--val_loss
s=0.1640-epoch=9-last.ckpt
parameters:
window_length_in_sec: 1.5 # Window length(s) in sec (floating-point number). Either a number or a list. Ex) 1.5 or [1.5,1.0,0.5]
shift_length_in_sec: 0.75 # Shift length(s) in sec (floating-point number). Either a number or a list. Ex) 0.75 or [0.75,0.5,0.25]
multiscale_weights: null # Weight for each scale. should be null (for single scale) or a list matched with window/shift scale count. Ex) [0.33,0.33,0.33]
save_embeddings: False # Save speaker embeddings in pickle format.
clustering:
parameters:
oracle_num_speakers: False # If True, use num of speakers value provided in manifest file.
max_num_speakers: 20 # Max number of speakers for each recording. If an oracle number of speakers is passed, this value is ignored.
enhanced_count_thres: 80 # If the number of segments is lower than this number, enhanced speaker counting is activated.
max_rp_threshold: 0.25 # Determines the range of p-value search: 0 < p <= max_rp_threshold. 0.25 0.10
sparse_search_volume: 30 # The higher the number, the more values will be examined with more time.
asr:
model_path: stt_en_conformer_ctc_large # Provide NGC cloud ASR model name. stt_en_conformer_ctc_* models are recommended for diarization purposes.
parameters:
asr_based_vad: True # if True, speech segmentation for diarization is based on word-timestamps from ASR inference.
asr_based_vad_threshold: 100 # Threshold (in sec) that caps the gap between two words when generating VAD timestamps using ASR based VAD.
asr_batch_size: null # Batch size can be dependent on each ASR model. Default batch sizes are applied if set to null.
lenient_overlap_WDER: True # If true, when a word falls into speaker-overlappedregions, consider the word as a correctly diarized word.
decoder_delay_in_sec: null # Native decoder delay. null is recommended to use the default values for each ASR model.
word_ts_anchor_offset: null # Offset to set a reference point from the start of the word. Recommended range of values is [-0.05 0.2]. null
word_ts_anchor_pos: "start" # Select which part of the word timestamp we want to use. The options are: 'start', 'end', 'mid'.
fix_word_ts_with_VAD: False # Fix the word timestamp using VAD output. You must provide a VAD model to use this feature.
colored_text: False # If True, use colored text to distinguish speakers in the output transcript.
print_time: True # If True, the start and end time of each speaker turn is printed in the output transcript.
break_lines: False # If True, the output transcript breaks the line to fix the line width (default is 90 chars)
ctc_decoder_parameters: # Optional beam search decoder (pyctcdecode)
pretrained_language_model: null # KenLM model file: .arpa model file or .bin binary file.
beam_width: 32
alpha: 0.5
beta: 2.5
realigning_lm_parameters: # Experimental feature
arpa_language_model: null # Provide a KenLM language model in .arpa format.
min_number_of_words: 3 # Min number of words for the left context.
max_number_of_words: 10 # Max number of words for the right context.
logprob_diff_threshold: 1.2 # The threshold for the difference between two log probability values from two hypotheses.
# json manifest line example
# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/file", "uem_filepathh
": "/path/to/uem/file"}
data preparation it segments all original audio into 3 seconds. (--create_segments)
import os
NEMO_ROOT = os.getcwd()
print(NEMO_ROOT)
import glob
import subprocess
import tarfile
import wget
# data_dir = os.path.join('/home/ec5017b/media-lab/nemo/NeMo/examples/speaker_tasks/diarization/','data')
data_dir='/workspace/nemo/examples/speaker_tasks/recognition/data'
nemo_root='/workspace/nemo'
wav_dir='/CN-celeb2-300/data/'
dest_dir='/CN-celeb2-300/'
os.system('find {}{} -iname "*.wav" > {}{}train_all.txt'.format(data_dir,wav_dir,data_dir,dest_dir))
os.system('python {}/scripts/speaker_tasks/filelist_to_manifest.py --create_segments --filelist {}{}train_all.txt --id -2 --out {}{}all_manifest.json --split'.format(nemo_root,dataa
_dir,dest_dir,data_dir,dest_dir))
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue was closed because it has been inactive for 7 days since being marked as stale.
Hi, I used pretrain titanet-large on one audio of Alimeeting_Eval_far (Chinese), and its Confusion Error Rate (CER) is 0.0283 So I think if fine-tune titanet-large with Chinese data (CN-celeb2) it will have a better result. However, the first result of CER is 0.1473 (learning rate: 1-e4), and the second one is 0.2120 (learning rate: 1-e6) Do you have some suggestions?
titanet-finetune.yaml
Add any other context about the problem here. GV100GL [Tesla V100 PCIe 16GB] (rev a1)