search

wenet-e2e / wespeaker

Research and Production Oriented Speaker Verification, Recognition and Diarization Toolkit
Apache License 2.0
707 stars 116 forks source link

KeyError: 'test/id00807-speech-01-001.wav' #118

Closed liyunlongaaa closed 1 year ago

liyunlongaaa commented 1 year ago

Screenshot from 2022-12-02 16-47-02

Hi, sorry to bother you, It seem apply cos score encounter erro，but I didn't know how to solve it. File format is made by .sh file，It seems that there should be no file mismatch,,,,,,

liyunlongaaa commented 1 year ago

It is note that Extract embeddings ouptut "Fail" and "Success"...maybe is key..

JiJiJiang commented 1 year ago

It is note that Extract embeddings ouptut "Fail" and "Success"...maybe is key..

Yeah ... check the embedding extracting log ...

liyunlongaaa commented 1 year ago

https://github.com/wenet-e2e/wespeaker/blob/7a7c3b11c3888d0e5bb7e64d3a50efa17ac56397/tools/extract_embedding.sh#L63-L67

but it seem the .sh is "if else"，why they will output together...

liyunlongaaa commented 1 year ago

https://github.com/wenet-e2e/wespeaker/blob/7a7c3b11c3888d0e5bb7e64d3a50efa17ac56397/tools/extract_embedding.sh#L63-L67

but it seem the .sh is "if else"，why they will output together...

oh I see !

liyunlongaaa commented 1 year ago

Let me reserch a reserch

liyunlongaaa commented 1 year ago

sorry, Why is it successful to extract eval embedding in the first stage of training (without LM), but it fails to extract eval embedding in the LM stage? Is there any difference between them?

JiJiJiang commented 1 year ago

It does not matter with LM. Could you paste the failure log?

liyunlongaaa commented 1 year ago

Thank you for your concern. In order to verify that the whole process can run smoothly, I re-run the script, and only trained 6 epochs in the first training, and only 2 trainings in the LM stage. Then this time they all had problems in the extraction eval stage, I was helpless.

liyunlongaaa commented 1 year ago

Preparing datasets ... Prepare wav.scp for each dataset ... Prepare train data including CN-Celeb_wav/dev and CN-Celeb2_wav ... Prepare data for testing ... Prepare data for enroll ... Prepare evalution trials ... Success !!! Now data preparation is done !!! Covert train and test data to raw... Start training ... [ INFO : 2022-12-03 01:00:31,481 ] - exp_dir is: exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6 [ INFO : 2022-12-03 01:00:31,481 ] - <== Passed Arguments ==> [ INFO : 2022-12-03 01:00:31,481 ] - {'data_type': 'raw', [ INFO : 2022-12-03 01:00:31,481 ] - 'dataloader_args': {'batch_size': 256, [ INFO : 2022-12-03 01:00:31,481 ] - 'drop_last': True, [ INFO : 2022-12-03 01:00:31,481 ] - 'num_workers': 16, [ INFO : 2022-12-03 01:00:31,481 ] - 'pin_memory': False, [ INFO : 2022-12-03 01:00:31,481 ] - 'prefetch_factor': 8}, [ INFO : 2022-12-03 01:00:31,481 ] - 'dataset_args': {'aug_prob': 0.6, [ INFO : 2022-12-03 01:00:31,481 ] - 'fbank_args': {'dither': 1.0, [ INFO : 2022-12-03 01:00:31,481 ] - 'frame_length': 25, [ INFO : 2022-12-03 01:00:31,481 ] - 'frame_shift': 10, [ INFO : 2022-12-03 01:00:31,481 ] - 'num_mel_bins': 80}, [ INFO : 2022-12-03 01:00:31,482 ] - 'num_frms': 200, [ INFO : 2022-12-03 01:00:31,482 ] - 'resample_rate': 16000, [ INFO : 2022-12-03 01:00:31,482 ] - 'shuffle': True, [ INFO : 2022-12-03 01:00:31,482 ] - 'shuffle_args': {'shuffle_size': 2500}, [ INFO : 2022-12-03 01:00:31,482 ] - 'spec_aug': False, [ INFO : 2022-12-03 01:00:31,482 ] - 'spec_aug_args': {'max_f': 8, [ INFO : 2022-12-03 01:00:31,482 ] - 'max_t': 10, [ INFO : 2022-12-03 01:00:31,482 ] - 'num_f_mask': 1, [ INFO : 2022-12-03 01:00:31,482 ] - 'num_t_mask': 1, [ INFO : 2022-12-03 01:00:31,482 ] - 'prob': 0.6}, [ INFO : 2022-12-03 01:00:31,482 ] - 'speed_perturb': True}, [ INFO : 2022-12-03 01:00:31,482 ] - 'enable_amp': False, [ INFO : 2022-12-03 01:00:31,482 ] - 'exp_dir': 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6', [ INFO : 2022-12-03 01:00:31,482 ] - 'gpus': [0], [ INFO : 2022-12-03 01:00:31,482 ] - 'log_batch_interval': 100, [ INFO : 2022-12-03 01:00:31,482 ] - 'loss': 'CrossEntropyLoss', [ INFO : 2022-12-03 01:00:31,482 ] - 'loss_args': {}, [ INFO : 2022-12-03 01:00:31,482 ] - 'margin_scheduler': 'MarginScheduler', [ INFO : 2022-12-03 01:00:31,482 ] - 'margin_update': {'final_margin': 0.2, [ INFO : 2022-12-03 01:00:31,482 ] - 'fix_start_epoch': 40, [ INFO : 2022-12-03 01:00:31,482 ] - 'increase_start_epoch': 20, [ INFO : 2022-12-03 01:00:31,482 ] - 'increase_type': 'exp', [ INFO : 2022-12-03 01:00:31,482 ] - 'initial_margin': 0.0, [ INFO : 2022-12-03 01:00:31,482 ] - 'update_margin': True}, [ INFO : 2022-12-03 01:00:31,482 ] - 'model': 'ResNet34', [ INFO : 2022-12-03 01:00:31,482 ] - 'model_args': {'embed_dim': 256, [ INFO : 2022-12-03 01:00:31,482 ] - 'feat_dim': 80, [ INFO : 2022-12-03 01:00:31,482 ] - 'pooling_func': 'TSTP', [ INFO : 2022-12-03 01:00:31,482 ] - 'two_emb_layer': False}, [ INFO : 2022-12-03 01:00:31,482 ] - 'model_init': None, [ INFO : 2022-12-03 01:00:31,482 ] - 'noise_data': '/home/yoos/Documents/data/musan/lmdb', [ INFO : 2022-12-03 01:00:31,482 ] - 'num_avg': 2, [ INFO : 2022-12-03 01:00:31,482 ] - 'num_epochs': 6, [ INFO : 2022-12-03 01:00:31,482 ] - 'optimizer': 'SGD', [ INFO : 2022-12-03 01:00:31,482 ] - 'optimizer_args': {'momentum': 0.9, 'nesterov': True, 'weight_decay': 0.0001}, [ INFO : 2022-12-03 01:00:31,482 ] - 'projection_args': {'easy_margin': False, [ INFO : 2022-12-03 01:00:31,482 ] - 'project_type': 'arc_margin', [ INFO : 2022-12-03 01:00:31,482 ] - 'scale': 32.0}, [ INFO : 2022-12-03 01:00:31,482 ] - 'reverb_data': '/home/yoos/Documents/data/rirs/lmdb', [ INFO : 2022-12-03 01:00:31,482 ] - 'save_epoch_interval': 5, [ INFO : 2022-12-03 01:00:31,482 ] - 'scheduler': 'ExponentialDecrease', [ INFO : 2022-12-03 01:00:31,482 ] - 'scheduler_args': {'final_lr': 5e-05, [ INFO : 2022-12-03 01:00:31,482 ] - 'initial_lr': 0.1, [ INFO : 2022-12-03 01:00:31,482 ] - 'warm_from_zero': True, [ INFO : 2022-12-03 01:00:31,482 ] - 'warm_up_epoch': 6}, [ INFO : 2022-12-03 01:00:31,482 ] - 'seed': 42, [ INFO : 2022-12-03 01:00:31,482 ] - 'train_data': '/home/yoos/Documents/data/cnceleb_train/raw.list', [ INFO : 2022-12-03 01:00:31,482 ] - 'train_label': '/home/yoos/Documents/data/cnceleb_train/utt2spk'} [ INFO : 2022-12-03 01:00:31,969 ] - <== Data statistics ==> [ INFO : 2022-12-03 01:00:31,969 ] - train data num: 519590, spk num: 2793 [ INFO : 2022-12-03 01:00:32,046 ] - <== Dataloaders ==> [ INFO : 2022-12-03 01:00:32,046 ] - train dataloaders created [ INFO : 2022-12-03 01:00:32,046 ] - loader size: 2029 [ INFO : 2022-12-03 01:00:32,046 ] - <== Model ==> [ INFO : 2022-12-03 01:00:32,122 ] - speaker_model size: 6634336 [ INFO : 2022-12-03 01:00:32,122 ] - Train model from scratch ... [ INFO : 2022-12-03 01:00:32,134 ] - ResNet( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (layer1): Sequential( [ INFO : 2022-12-03 01:00:32,134 ] - (0): BasicBlock( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - (1): BasicBlock( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - (2): BasicBlock( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - (layer2): Sequential( [ INFO : 2022-12-03 01:00:32,134 ] - (0): BasicBlock( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (shortcut): Sequential( [ INFO : 2022-12-03 01:00:32,134 ] - (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - (1): BasicBlock( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,134 ] - ) [ INFO : 2022-12-03 01:00:32,134 ] - (2): BasicBlock( [ INFO : 2022-12-03 01:00:32,134 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,134 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,134 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (3): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (layer3): Sequential( [ INFO : 2022-12-03 01:00:32,135 ] - (0): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential( [ INFO : 2022-12-03 01:00:32,135 ] - (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (1): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (2): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (3): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (4): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (5): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,135 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,135 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - ) [ INFO : 2022-12-03 01:00:32,135 ] - (layer4): Sequential( [ INFO : 2022-12-03 01:00:32,135 ] - (0): BasicBlock( [ INFO : 2022-12-03 01:00:32,135 ] - (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - (shortcut): Sequential( [ INFO : 2022-12-03 01:00:32,136 ] - (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,136 ] - (1): BasicBlock( [ INFO : 2022-12-03 01:00:32,136 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,136 ] - (2): BasicBlock( [ INFO : 2022-12-03 01:00:32,136 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 01:00:32,136 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 01:00:32,136 ] - (shortcut): Sequential() [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,136 ] - (pool): TSTP() [ INFO : 2022-12-03 01:00:32,136 ] - (seg_1): Linear(in_features=5120, out_features=256, bias=True) [ INFO : 2022-12-03 01:00:32,136 ] - (seg_bn_1): Identity() [ INFO : 2022-12-03 01:00:32,136 ] - (seg_2): Identity() [ INFO : 2022-12-03 01:00:32,136 ] - (projection): ArcMarginProduct( [ INFO : 2022-12-03 01:00:32,136 ] - in_features=256, out_features=8379, scale=32.0, [ INFO : 2022-12-03 01:00:32,136 ] - margin=0.0, easy_margin=False [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,136 ] - ) [ INFO : 2022-12-03 01:00:32,410 ] - start_epoch: 1 [ INFO : 2022-12-03 01:00:32,420 ] - <== Loss ==> [ INFO : 2022-12-03 01:00:32,420 ] - loss criterion is: CrossEntropyLoss [ INFO : 2022-12-03 01:00:32,420 ] - <== Optimizer ==> [ INFO : 2022-12-03 01:00:32,420 ] - optimizer is: SGD [ INFO : 2022-12-03 01:00:32,420 ] - <== Scheduler ==> [ INFO : 2022-12-03 01:00:32,420 ] - scheduler is: ExponentialDecrease [ INFO : 2022-12-03 01:00:32,421 ] - <== MarginScheduler ==> [ INFO : 2022-12-03 01:00:32,422 ] - <========== Training process ==========> [ INFO : 2022-12-03 01:00:32,422 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 01:00:32,423 ] - | Epoch| Batch| Lr| Margin| Loss| Acc| [ INFO : 2022-12-03 01:00:32,423 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 01:00:46,074 ] - Reducer buckets have been rebuilt in this iteration. [ INFO : 2022-12-03 01:01:37,963 ] - | 1| 100| 0.0030579| 0| 9.5142| 0.11719| [ INFO : 2022-12-03 01:02:30,460 ] - | 1| 200| 0.0057746| 0| 9.3304| 0.15234| [ INFO : 2022-12-03 01:03:21,603 ] - | 1| 300| 0.0081512| 0| 9.294| 0.1849| [ INFO : 2022-12-03 01:04:12,906 ] - | 1| 400| 0.010219| 0| 9.2092| 0.26953| [ INFO : 2022-12-03 01:05:04,002 ] - | 1| 500| 0.012007| 0| 9.079| 0.43672| [ INFO : 2022-12-03 01:05:55,343 ] - | 1| 600| 0.01354| 0| 8.9016| 0.74935| [ INFO : 2022-12-03 01:06:46,647 ] - | 1| 700| 0.014845| 0| 8.6881| 1.1568| [ INFO : 2022-12-03 01:07:37,732 ] - | 1| 800| 0.015941| 0| 8.4551| 1.8032| [ INFO : 2022-12-03 01:08:29,148 ] - | 1| 900| 0.016851| 0| 8.216| 2.6393| [ INFO : 2022-12-03 01:09:20,491 ] - | 1| 1000| 0.017592| 0| 7.9755| 3.6691| [ INFO : 2022-12-03 01:10:11,564 ] - | 1| 1100| 0.018181| 0| 7.7428| 4.8168| [ INFO : 2022-12-03 01:11:02,888 ] - | 1| 1200| 0.018635| 0| 7.5176| 6.0501| [ INFO : 2022-12-03 01:11:54,175 ] - | 1| 1300| 0.018967| 0| 7.3034| 7.3468| [ INFO : 2022-12-03 01:12:45,351 ] - | 1| 1400| 0.019191| 0| 7.0991| 8.7079| [ INFO : 2022-12-03 01:13:36,718 ] - | 1| 1500| 0.019318| 0| 6.9121| 10.009| [ INFO : 2022-12-03 01:14:27,865 ] - | 1| 1600| 0.01936| 0| 6.7324| 11.345| [ INFO : 2022-12-03 01:15:19,169 ] - | 1| 1700| 0.019325| 0| 6.5631| 12.673| [ INFO : 2022-12-03 01:16:10,515 ] - | 1| 1800| 0.019224| 0| 6.404| 13.945| [ INFO : 2022-12-03 01:17:01,597 ] - | 1| 1900| 0.019065| 0| 6.2542| 15.198| [ INFO : 2022-12-03 01:17:52,082 ] - | 1| 2000| 0.018854| 0| 6.1146| 16.4| [ INFO : 2022-12-03 01:18:00,444 ] - | 1| 2016| 0.018816| 0| 6.0926| 16.595| [ INFO : 2022-12-03 01:19:02,188 ] - | 2| 100| 0.018518| 0| 3.2682| 41.699| [ INFO : 2022-12-03 01:19:54,676 ] - | 2| 200| 0.018214| 0| 3.2327| 42.465| [ INFO : 2022-12-03 01:20:45,859 ] - | 2| 300| 0.01788| 0| 3.1874| 43.059| [ INFO : 2022-12-03 01:21:37,244 ] - | 2| 400| 0.017519| 0| 3.137| 43.855| [ INFO : 2022-12-03 01:22:28,315 ] - | 2| 500| 0.017137| 0| 3.0917| 44.545| [ INFO : 2022-12-03 01:23:19,624 ] - | 2| 600| 0.016736| 0| 3.0502| 45.173| [ INFO : 2022-12-03 01:24:11,010 ] - | 2| 700| 0.016322| 0| 3.0144| 45.731| [ INFO : 2022-12-03 01:25:02,108 ] - | 2| 800| 0.015896| 0| 2.9739| 46.36| [ INFO : 2022-12-03 01:25:53,435 ] - | 2| 900| 0.015462| 0| 2.9347| 46.975| [ INFO : 2022-12-03 01:26:44,819 ] - | 2| 1000| 0.015022| 0| 2.9013| 47.508| [ INFO : 2022-12-03 01:27:35,949 ] - | 2| 1100| 0.014579| 0| 2.8674| 48.044| [ INFO : 2022-12-03 01:28:27,290 ] - | 2| 1200| 0.014134| 0| 2.8337| 48.616| [ INFO : 2022-12-03 01:29:18,582 ] - | 2| 1300| 0.01369| 0| 2.8027| 49.127| [ INFO : 2022-12-03 01:30:09,646 ] - | 2| 1400| 0.013248| 0| 2.772| 49.648| [ INFO : 2022-12-03 01:31:00,963 ] - | 2| 1500| 0.012809| 0| 2.7433| 50.107| [ INFO : 2022-12-03 01:31:52,083 ] - | 2| 1600| 0.012375| 0| 2.7157| 50.564| [ INFO : 2022-12-03 01:32:43,329 ] - | 2| 1700| 0.011946| 0| 2.6884| 51.007| [ INFO : 2022-12-03 01:33:34,604 ] - | 2| 1800| 0.011524| 0| 2.6608| 51.464| [ INFO : 2022-12-03 01:34:25,705 ] - | 2| 1900| 0.01111| 0| 2.6345| 51.904| [ INFO : 2022-12-03 01:35:16,208 ] - | 2| 2000| 0.010703| 0| 2.6112| 52.283| [ INFO : 2022-12-03 01:35:24,566 ] - | 2| 2016| 0.010639| 0| 2.6075| 52.341| [ INFO : 2022-12-03 01:36:27,487 ] - | 3| 100| 0.010191| 0| 2.0719| 60.973| [ INFO : 2022-12-03 01:37:19,817 ] - | 3| 200| 0.0098045| 0| 2.0588| 61.096| [ INFO : 2022-12-03 01:38:10,974 ] - | 3| 300| 0.0094275| 0| 2.0516| 61.328| [ INFO : 2022-12-03 01:39:02,306 ] - | 3| 400| 0.0090602| 0| 2.0358| 61.663| [ INFO : 2022-12-03 01:39:53,405 ] - | 3| 500| 0.0087027| 0| 2.0186| 61.976| [ INFO : 2022-12-03 01:40:44,760 ] - | 3| 600| 0.0083554| 0| 2.0117| 62.118| [ INFO : 2022-12-03 01:41:36,076 ] - | 3| 700| 0.0080183| 0| 1.9992| 62.345| [ INFO : 2022-12-03 01:42:27,158 ] - | 3| 800| 0.0076913| 0| 1.9914| 62.479| [ INFO : 2022-12-03 01:43:18,454 ] - | 3| 900| 0.0073745| 0| 1.9811| 62.674| [ INFO : 2022-12-03 01:44:09,839 ] - | 3| 1000| 0.0070679| 0| 1.9697| 62.87| [ INFO : 2022-12-03 01:45:01,019 ] - | 3| 1100| 0.0067715| 0| 1.9588| 63.08| [ INFO : 2022-12-03 01:45:52,399 ] - | 3| 1200| 0.006485| 0| 1.9517| 63.211| [ INFO : 2022-12-03 01:46:43,692 ] - | 3| 1300| 0.0062083| 0| 1.9404| 63.41| [ INFO : 2022-12-03 01:47:34,875 ] - | 3| 1400| 0.0059415| 0| 1.9295| 63.613| [ INFO : 2022-12-03 01:48:26,163 ] - | 3| 1500| 0.0056841| 0| 1.9191| 63.799| [ INFO : 2022-12-03 01:49:17,371 ] - | 3| 1600| 0.0054362| 0| 1.9113| 63.962| [ INFO : 2022-12-03 01:50:08,602 ] - | 3| 1700| 0.0051974| 0| 1.9019| 64.141| [ INFO : 2022-12-03 01:50:59,880 ] - | 3| 1800| 0.0049677| 0| 1.8936| 64.307| [ INFO : 2022-12-03 01:51:50,884 ] - | 3| 1900| 0.0047467| 0| 1.8869| 64.438| [ INFO : 2022-12-03 01:52:41,368 ] - | 3| 2000| 0.0045342| 0| 1.8785| 64.612| [ INFO : 2022-12-03 01:52:49,736 ] - | 3| 2016| 0.004501| 0| 1.8763| 64.647| [ INFO : 2022-12-03 01:53:51,993 ] - | 4| 100| 0.0042725| 0| 1.66| 68.27| [ INFO : 2022-12-03 01:54:44,457 ] - | 4| 200| 0.0040787| 0| 1.6715| 68.031| [ INFO : 2022-12-03 01:55:35,629 ] - | 4| 300| 0.0038928| 0| 1.6694| 68.164| [ INFO : 2022-12-03 01:56:26,931 ] - | 4| 400| 0.0037145| 0| 1.661| 68.354| [ INFO : 2022-12-03 01:57:18,045 ] - | 4| 500| 0.0035435| 0| 1.656| 68.443| [ INFO : 2022-12-03 01:58:09,385 ] - | 4| 600| 0.0033795| 0| 1.6529| 68.595| [ INFO : 2022-12-03 01:59:00,758 ] - | 4| 700| 0.0032225| 0| 1.6503| 68.662| [ INFO : 2022-12-03 01:59:51,819 ] - | 4| 800| 0.003072| 0| 1.6495| 68.682| [ INFO : 2022-12-03 02:00:43,139 ] - | 4| 900| 0.002928| 0| 1.6468| 68.754| [ INFO : 2022-12-03 02:01:34,505 ] - | 4| 1000| 0.0027902| 0| 1.6412| 68.873| [ INFO : 2022-12-03 02:02:25,597 ] - | 4| 1100| 0.0026583| 0| 1.6359| 68.963| [ INFO : 2022-12-03 02:03:17,058 ] - | 4| 1200| 0.0025321| 0| 1.6317| 69.055| [ INFO : 2022-12-03 02:04:08,400 ] - | 4| 1300| 0.0024115| 0| 1.629| 69.129| [ INFO : 2022-12-03 02:04:59,484 ] - | 4| 1400| 0.0022962| 0| 1.6249| 69.208| [ INFO : 2022-12-03 02:05:50,800 ] - | 4| 1500| 0.0021861| 0| 1.6228| 69.271| [ INFO : 2022-12-03 02:06:41,913 ] - | 4| 1600| 0.0020808| 0| 1.618| 69.359| [ INFO : 2022-12-03 02:07:33,242 ] - | 4| 1700| 0.0019803| 0| 1.6143| 69.44| [ INFO : 2022-12-03 02:08:24,553 ] - | 4| 1800| 0.0018844| 0| 1.6126| 69.504| [ INFO : 2022-12-03 02:09:15,598 ] - | 4| 1900| 0.0017927| 0| 1.6088| 69.577| [ INFO : 2022-12-03 02:10:06,053 ] - | 4| 2000| 0.0017053| 0| 1.6046| 69.666| [ INFO : 2022-12-03 02:10:14,415 ] - | 4| 2016| 0.0016917| 0| 1.6042| 69.674| [ INFO : 2022-12-03 02:11:16,613 ] - | 5| 100| 0.0015985| 0| 1.5455| 70.715| [ INFO : 2022-12-03 02:12:09,077 ] - | 5| 200| 0.00152| 0| 1.5492| 70.646| [ INFO : 2022-12-03 02:13:00,238 ] - | 5| 300| 0.0014452| 0| 1.5444| 70.784| [ INFO : 2022-12-03 02:13:51,580 ] - | 5| 400| 0.0013738| 0| 1.536| 70.99| [ INFO : 2022-12-03 02:14:42,665 ] - | 5| 500| 0.0013058| 0| 1.533| 71.066| [ INFO : 2022-12-03 02:15:34,001 ] - | 5| 600| 0.001241| 0| 1.5275| 71.176| [ INFO : 2022-12-03 02:16:25,308 ] - | 5| 700| 0.0011793| 0| 1.5256| 71.219| [ INFO : 2022-12-03 02:17:16,473 ] - | 5| 800| 0.0011205| 0| 1.5217| 71.229| [ INFO : 2022-12-03 02:18:07,716 ] - | 5| 900| 0.0010645| 0| 1.5174| 71.333| [ INFO : 2022-12-03 02:18:59,053 ] - | 5| 1000| 0.0010111| 0| 1.5115| 71.433| [ INFO : 2022-12-03 02:19:50,090 ] - | 5| 1100|0.00096037| 0| 1.5098| 71.462| [ INFO : 2022-12-03 02:20:41,329 ] - | 5| 1200|0.00091203| 0| 1.5061| 71.548| [ INFO : 2022-12-03 02:21:32,595 ] - | 5| 1300|0.00086603| 0| 1.5033| 71.611| [ INFO : 2022-12-03 02:22:23,702 ] - | 5| 1400|0.00082225| 0| 1.5001| 71.646| [ INFO : 2022-12-03 02:23:15,028 ] - | 5| 1500| 0.0007806| 0| 1.4983| 71.666| [ INFO : 2022-12-03 02:24:06,203 ] - | 5| 1600|0.00074098| 0| 1.4958| 71.714| [ INFO : 2022-12-03 02:24:57,441 ] - | 5| 1700| 0.0007033| 0| 1.4946| 71.72| [ INFO : 2022-12-03 02:25:48,705 ] - | 5| 1800|0.00066746| 0| 1.4937| 71.718| [ INFO : 2022-12-03 02:26:39,789 ] - | 5| 1900|0.00063339| 0| 1.4915| 71.762| [ INFO : 2022-12-03 02:27:30,228 ] - | 5| 2000|0.00060099| 0| 1.4908| 71.767| [ INFO : 2022-12-03 02:27:38,574 ] - | 5| 2016|0.00059596| 0| 1.4909| 71.766| [ INFO : 2022-12-03 02:28:40,379 ] - | 6| 100|0.00056156| 0| 1.4484| 72.684| [ INFO : 2022-12-03 02:29:32,842 ] - | 6| 200|0.00053272| 0| 1.4583| 72.447| [ INFO : 2022-12-03 02:30:23,985 ] - | 6| 300|0.00050531| 0| 1.4556| 72.465| [ INFO : 2022-12-03 02:31:15,241 ] - | 6| 400|0.00047927| 0| 1.4564| 72.448| [ INFO : 2022-12-03 02:32:06,305 ] - | 6| 500|0.00045454| 0| 1.4566| 72.499| [ INFO : 2022-12-03 02:32:57,650 ] - | 6| 600|0.00043104| 0| 1.4595| 72.471| [ INFO : 2022-12-03 02:33:48,963 ] - | 6| 700|0.00040872| 0| 1.4595| 72.474| [ INFO : 2022-12-03 02:34:40,021 ] - | 6| 800|0.00038752| 0| 1.4566| 72.477| [ INFO : 2022-12-03 02:35:31,300 ] - | 6| 900|0.00036739| 0| 1.4509| 72.57| [ INFO : 2022-12-03 02:36:22,623 ] - | 6| 1000|0.00034828| 0| 1.4483| 72.57| [ INFO : 2022-12-03 02:37:13,663 ] - | 6| 1100|0.00033013| 0| 1.4503| 72.549| [ INFO : 2022-12-03 02:38:04,983 ] - | 6| 1200|0.00031291| 0| 1.449| 72.59| [ INFO : 2022-12-03 02:38:56,273 ] - | 6| 1300|0.00029656| 0| 1.4494| 72.586| [ INFO : 2022-12-03 02:39:47,406 ] - | 6| 1400|0.00028105| 0| 1.4498| 72.564| [ INFO : 2022-12-03 02:40:38,807 ] - | 6| 1500|0.00026632| 0| 1.4479| 72.603| [ INFO : 2022-12-03 02:41:29,931 ] - | 6| 1600|0.00025235| 0| 1.4492| 72.579| [ INFO : 2022-12-03 02:42:21,223 ] - | 6| 1700| 0.0002391| 0| 1.4483| 72.625| [ INFO : 2022-12-03 02:43:12,508 ] - | 6| 1800|0.00022652| 0| 1.4471| 72.641| [ INFO : 2022-12-03 02:44:03,550 ] - | 6| 1900|0.00021459| 0| 1.4464| 72.655| [ INFO : 2022-12-03 02:44:54,014 ] - | 6| 2000|0.00020328| 0| 1.4468| 72.662| [ INFO : 2022-12-03 02:45:02,359 ] - | 6| 2016|0.00020152| 0| 1.4464| 72.674| [ INFO : 2022-12-03 02:45:02,428 ] - +----------+----------+----------+----------+----------+----------+ Namespace(dst_model='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/avg_model.pt', src_path='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models', num=2, min_epoch=0, max_epoch=65536) ['exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/model_5.pt', 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/model_6.pt'] Processing exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/model_5.pt Processing exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/model_6.pt Saving to exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/avg_model.pt Extract embeddings ... extract_embedding from /home/yoos/Documents/data/cnceleb_train/raw.list, wavs_num: 519590 extract_embedding from /home/yoos/Documents/data/eval/raw.list, wavs_num: 18772 Fail eval Success cnceleb_train Embedding dir is (exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/embeddings). mean vector of enroll 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [00:00<00:00, 70239.54it/s] Score ... apply cosine scoring ... CNC-Eval-Concat.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Concat.lst: 0%| | 4884/3484292 [00:00<02:59, 19340.65it/s] Traceback (most recent call last): File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 96, in fire.Fire(main) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, kwargs) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 92, in main trials_cosine_score(eval_scp_path, store_score_dir, mean_vec_path, trials) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 62, in trials_cosine_score emb1, emb2 = emb_dict[segs[0]], emb_dict[segs[1]] KeyError: 'test/id00856-speech-01-001.wav' CNC-Eval-Avg.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Avg.lst: 0%| | 4884/3484292 [00:00<03:07, 18519.41it/s] Traceback (most recent call last): File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 96, in fire.Fire(main) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 92, in main trials_cosine_score(eval_scp_path, store_score_dir, mean_vec_path, trials) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 62, in trials_cosine_score emb1, emb2 = emb_dict[segs[0]], emb_dict[segs[1]] KeyError: 'test/id00856-speech-01-001.wav' compute metrics (EER/minDCF) ... ---- CNC-Eval-Concat.lst.score ----- EER = 21.996 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.467 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/scores/CNC-Eval-Concat.lst.score.det.png ---- CNC-Eval-Avg.lst.score ----- EER = 20.000 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.467 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/scores/CNC-Eval-Avg.lst.score.det.png Score norm ... compute mean xvector 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2793/2793 [00:00<00:00, 9950.46it/s] compute norm score 2022-12-03 02:52:30,102 INFO get embedding ... 2022-12-03 02:52:30,222 INFO computing normed score ... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4884/4884 [00:00<00:00, 938171.78it/s] 2022-12-03 02:52:30,756 INFO Over! 2022-12-03 02:52:30,872 INFO get embedding ... 2022-12-03 02:52:31,007 INFO computing normed score ... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4884/4884 [00:00<00:00, 928729.23it/s] 2022-12-03 02:52:31,584 INFO Over! compute metrics ---- cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score ----- EER = 20.559 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.467 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/scores/cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score.det.png ---- cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score ----- EER = 20.000 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.467 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/scores/cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score.det.png Export the best model ... ResNet( (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (layer1): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (1): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer2): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer3): Sequential( (0): BasicBlock( (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (4): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (5): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer4): Sequential( (0): BasicBlock( (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (pool): TSTP() (seg_1): Linear(in_features=5120, out_features=256, bias=True) (seg_bn_1): Identity() (seg_2): Identity() ) Export model successfully, see exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6/models/final.zip Large margin fine-tuning ... Start training ... exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models already exists !!! [ INFO : 2022-12-03 02:52:38,763 ] - exp_dir is: exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM [ INFO : 2022-12-03 02:52:38,764 ] - <== Passed Arguments ==> [ INFO : 2022-12-03 02:52:38,764 ] - {'checkpoint': 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/model_0.pt', [ INFO : 2022-12-03 02:52:38,764 ] - 'data_type': 'raw', [ INFO : 2022-12-03 02:52:38,764 ] - 'dataloader_args': {'batch_size': 64, [ INFO : 2022-12-03 02:52:38,764 ] - 'drop_last': True, [ INFO : 2022-12-03 02:52:38,764 ] - 'num_workers': 16, [ INFO : 2022-12-03 02:52:38,764 ] - 'pin_memory': False, [ INFO : 2022-12-03 02:52:38,764 ] - 'prefetch_factor': 8}, [ INFO : 2022-12-03 02:52:38,764 ] - 'dataset_args': {'aug_prob': 0.6, [ INFO : 2022-12-03 02:52:38,764 ] - 'fbank_args': {'dither': 1.0, [ INFO : 2022-12-03 02:52:38,764 ] - 'frame_length': 25, [ INFO : 2022-12-03 02:52:38,764 ] - 'frame_shift': 10, [ INFO : 2022-12-03 02:52:38,764 ] - 'num_mel_bins': 80}, [ INFO : 2022-12-03 02:52:38,764 ] - 'num_frms': 600, [ INFO : 2022-12-03 02:52:38,764 ] - 'resample_rate': 16000, [ INFO : 2022-12-03 02:52:38,764 ] - 'shuffle': True, [ INFO : 2022-12-03 02:52:38,764 ] - 'shuffle_args': {'shuffle_size': 2500}, [ INFO : 2022-12-03 02:52:38,764 ] - 'spec_aug': False, [ INFO : 2022-12-03 02:52:38,764 ] - 'spec_aug_args': {'max_f': 8, [ INFO : 2022-12-03 02:52:38,764 ] - 'max_t': 10, [ INFO : 2022-12-03 02:52:38,764 ] - 'num_f_mask': 1, [ INFO : 2022-12-03 02:52:38,764 ] - 'num_t_mask': 1, [ INFO : 2022-12-03 02:52:38,764 ] - 'prob': 0.6}, [ INFO : 2022-12-03 02:52:38,764 ] - 'speed_perturb': True}, [ INFO : 2022-12-03 02:52:38,764 ] - 'do_lm': True, [ INFO : 2022-12-03 02:52:38,764 ] - 'enable_amp': False, [ INFO : 2022-12-03 02:52:38,764 ] - 'exp_dir': 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM', [ INFO : 2022-12-03 02:52:38,764 ] - 'gpus': [0], [ INFO : 2022-12-03 02:52:38,764 ] - 'log_batch_interval': 100, [ INFO : 2022-12-03 02:52:38,764 ] - 'loss': 'CrossEntropyLoss', [ INFO : 2022-12-03 02:52:38,764 ] - 'loss_args': {}, [ INFO : 2022-12-03 02:52:38,764 ] - 'margin_scheduler': 'MarginScheduler', [ INFO : 2022-12-03 02:52:38,764 ] - 'margin_update': {'final_margin': 0.5, [ INFO : 2022-12-03 02:52:38,764 ] - 'fix_start_epoch': 1, [ INFO : 2022-12-03 02:52:38,764 ] - 'increase_start_epoch': 1, [ INFO : 2022-12-03 02:52:38,764 ] - 'increase_type': 'exp', [ INFO : 2022-12-03 02:52:38,764 ] - 'initial_margin': 0.5, [ INFO : 2022-12-03 02:52:38,764 ] - 'update_margin': True}, [ INFO : 2022-12-03 02:52:38,764 ] - 'model': 'ResNet34', [ INFO : 2022-12-03 02:52:38,764 ] - 'model_args': {'embed_dim': 256, [ INFO : 2022-12-03 02:52:38,764 ] - 'feat_dim': 80, [ INFO : 2022-12-03 02:52:38,764 ] - 'pooling_func': 'TSTP', [ INFO : 2022-12-03 02:52:38,764 ] - 'two_emb_layer': False}, [ INFO : 2022-12-03 02:52:38,764 ] - 'model_init': None, [ INFO : 2022-12-03 02:52:38,764 ] - 'noise_data': '/home/yoos/Documents/data/musan/lmdb', [ INFO : 2022-12-03 02:52:38,764 ] - 'num_avg': 1, [ INFO : 2022-12-03 02:52:38,764 ] - 'num_epochs': 2, [ INFO : 2022-12-03 02:52:38,764 ] - 'optimizer': 'SGD', [ INFO : 2022-12-03 02:52:38,764 ] - 'optimizer_args': {'momentum': 0.9, 'nesterov': True, 'weight_decay': 0.0001}, [ INFO : 2022-12-03 02:52:38,765 ] - 'projection_args': {'easy_margin': False, [ INFO : 2022-12-03 02:52:38,765 ] - 'project_type': 'arc_margin', [ INFO : 2022-12-03 02:52:38,765 ] - 'scale': 32.0}, [ INFO : 2022-12-03 02:52:38,765 ] - 'reverb_data': '/home/yoos/Documents/data/rirs/lmdb', [ INFO : 2022-12-03 02:52:38,765 ] - 'save_epoch_interval': 1, [ INFO : 2022-12-03 02:52:38,765 ] - 'scheduler': 'ExponentialDecrease', [ INFO : 2022-12-03 02:52:38,765 ] - 'scheduler_args': {'final_lr': 2.5e-05, [ INFO : 2022-12-03 02:52:38,765 ] - 'initial_lr': 0.0001, [ INFO : 2022-12-03 02:52:38,765 ] - 'warm_from_zero': True, [ INFO : 2022-12-03 02:52:38,765 ] - 'warm_up_epoch': 1}, [ INFO : 2022-12-03 02:52:38,765 ] - 'seed': 42, [ INFO : 2022-12-03 02:52:38,765 ] - 'train_data': '/home/yoos/Documents/data/cnceleb_train/raw.list', [ INFO : 2022-12-03 02:52:38,765 ] - 'train_label': '/home/yoos/Documents/data/cnceleb_train/utt2spk'} [ INFO : 2022-12-03 02:52:39,249 ] - <== Data statistics ==> [ INFO : 2022-12-03 02:52:39,249 ] - train data num: 519590, spk num: 2793 [ INFO : 2022-12-03 02:52:39,337 ] - <== Dataloaders ==> [ INFO : 2022-12-03 02:52:39,337 ] - train dataloaders created [ INFO : 2022-12-03 02:52:39,337 ] - loader size: 8118 [ INFO : 2022-12-03 02:52:39,337 ] - <== Model ==> [ INFO : 2022-12-03 02:52:39,391 ] - speaker_model size: 6634336 [ INFO : 2022-12-03 02:52:39,391 ] - No speed perturb while doing large margin fine-tuning [ INFO : 2022-12-03 02:52:39,398 ] - ResNet( [ INFO : 2022-12-03 02:52:39,398 ] - (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,398 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,398 ] - (layer1): Sequential( [ INFO : 2022-12-03 02:52:39,399 ] - (0): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (1): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (2): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (layer2): Sequential( [ INFO : 2022-12-03 02:52:39,399 ] - (0): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential( [ INFO : 2022-12-03 02:52:39,399 ] - (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (1): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (2): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (3): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - ) [ INFO : 2022-12-03 02:52:39,399 ] - (layer3): Sequential( [ INFO : 2022-12-03 02:52:39,399 ] - (0): BasicBlock( [ INFO : 2022-12-03 02:52:39,399 ] - (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,399 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,399 ] - (shortcut): Sequential( [ INFO : 2022-12-03 02:52:39,399 ] - (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (1): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (2): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (3): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (4): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (5): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (layer4): Sequential( [ INFO : 2022-12-03 02:52:39,400 ] - (0): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential( [ INFO : 2022-12-03 02:52:39,400 ] - (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (1): BasicBlock( [ INFO : 2022-12-03 02:52:39,400 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,400 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,400 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,400 ] - ) [ INFO : 2022-12-03 02:52:39,400 ] - (2): BasicBlock( [ INFO : 2022-12-03 02:52:39,401 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,401 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,401 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 02:52:39,401 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 02:52:39,401 ] - (shortcut): Sequential() [ INFO : 2022-12-03 02:52:39,401 ] - ) [ INFO : 2022-12-03 02:52:39,401 ] - ) [ INFO : 2022-12-03 02:52:39,401 ] - (pool): TSTP() [ INFO : 2022-12-03 02:52:39,401 ] - (seg_1): Linear(in_features=5120, out_features=256, bias=True) [ INFO : 2022-12-03 02:52:39,401 ] - (seg_bn_1): Identity() [ INFO : 2022-12-03 02:52:39,401 ] - (seg_2): Identity() [ INFO : 2022-12-03 02:52:39,401 ] - (projection): ArcMarginProduct( [ INFO : 2022-12-03 02:52:39,401 ] - in_features=256, out_features=8379, scale=32.0, [ INFO : 2022-12-03 02:52:39,401 ] - margin=0.0, easy_margin=False [ INFO : 2022-12-03 02:52:39,401 ] - ) [ INFO : 2022-12-03 02:52:39,401 ] - ) [ INFO : 2022-12-03 02:52:39,595 ] - Load checkpoint: exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/model_0.pt [ INFO : 2022-12-03 02:52:39,596 ] - start_epoch: 1 [ INFO : 2022-12-03 02:52:39,608 ] - <== Loss ==> [ INFO : 2022-12-03 02:52:39,608 ] - loss criterion is: CrossEntropyLoss [ INFO : 2022-12-03 02:52:39,608 ] - <== Optimizer ==> [ INFO : 2022-12-03 02:52:39,608 ] - optimizer is: SGD [ INFO : 2022-12-03 02:52:39,608 ] - <== Scheduler ==> [ INFO : 2022-12-03 02:52:39,608 ] - scheduler is: ExponentialDecrease [ INFO : 2022-12-03 02:52:39,608 ] - <== MarginScheduler ==> [ INFO : 2022-12-03 02:52:39,610 ] - <========== Training process ==========> [ INFO : 2022-12-03 02:52:39,611 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 02:52:39,611 ] - | Epoch| Batch| Lr| Margin| Loss| Acc| [ INFO : 2022-12-03 02:52:39,611 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 02:52:49,511 ] - Reducer buckets have been rebuilt in this iteration. [ INFO : 2022-12-03 02:53:26,758 ] - | 1| 100|1.2092e-06| 0.5| 12.487| 0.015625| [ INFO : 2022-12-03 02:54:03,939 ] - | 1| 200| 2.41e-06| 0.5| 12.487| 0.015625| [ INFO : 2022-12-03 02:54:41,101 ] - | 1| 300|3.5903e-06| 0.5| 12.468| 0.010417| [ INFO : 2022-12-03 02:55:18,268 ] - | 1| 400|4.7504e-06| 0.5| 12.455| 0.011719| [ INFO : 2022-12-03 02:55:55,434 ] - | 1| 500|5.8904e-06| 0.5| 12.451| 0.009375| [ INFO : 2022-12-03 02:56:32,782 ] - | 1| 600|7.0108e-06| 0.5| 12.443| 0.010417| [ INFO : 2022-12-03 02:57:09,951 ] - | 1| 700|8.1116e-06| 0.5| 12.436| 0.0089286| [ INFO : 2022-12-03 02:57:47,101 ] - | 1| 800|9.1933e-06| 0.5| 12.437| 0.0078125| [ INFO : 2022-12-03 02:58:24,249 ] - | 1| 900|1.0256e-05| 0.5| 12.43| 0.0069444| [ INFO : 2022-12-03 02:59:01,399 ] - | 1| 1000| 1.13e-05| 0.5| 12.425| 0.0078125| [ INFO : 2022-12-03 02:59:38,544 ] - | 1| 1100|1.2325e-05| 0.5| 12.414| 0.0085227| [ INFO : 2022-12-03 03:00:15,873 ] - | 1| 1200|1.3332e-05| 0.5| 12.408| 0.0078125| [ INFO : 2022-12-03 03:00:53,040 ] - | 1| 1300|1.4322e-05| 0.5| 12.407| 0.0096154| [ INFO : 2022-12-03 03:01:30,171 ] - | 1| 1400|1.5293e-05| 0.5| 12.4| 0.011161| [ INFO : 2022-12-03 03:02:07,304 ] - | 1| 1500|1.6247e-05| 0.5| 12.398| 0.011458| [ INFO : 2022-12-03 03:02:44,420 ] - | 1| 1600|1.7183e-05| 0.5| 12.392| 0.011719| [ INFO : 2022-12-03 03:03:21,544 ] - | 1| 1700|1.8103e-05| 0.5| 12.386| 0.011949| [ INFO : 2022-12-03 03:03:58,821 ] - | 1| 1800|1.9005e-05| 0.5| 12.381| 0.011285| [ INFO : 2022-12-03 03:04:35,956 ] - | 1| 1900|1.9891e-05| 0.5| 12.374| 0.011513| [ INFO : 2022-12-03 03:05:13,083 ] - | 1| 2000| 2.076e-05| 0.5| 12.371| 0.0125| [ INFO : 2022-12-03 03:05:50,217 ] - | 1| 2100|2.1614e-05| 0.5| 12.37| 0.013393| [ INFO : 2022-12-03 03:06:27,374 ] - | 1| 2200|2.2451e-05| 0.5| 12.367| 0.014915| [ INFO : 2022-12-03 03:07:04,495 ] - | 1| 2300|2.3272e-05| 0.5| 12.366| 0.014946| [ INFO : 2022-12-03 03:07:41,771 ] - | 1| 2400|2.4078e-05| 0.5| 12.365| 0.014974| [ INFO : 2022-12-03 03:08:18,904 ] - | 1| 2500|2.4869e-05| 0.5| 12.362| 0.015| [ INFO : 2022-12-03 03:08:56,026 ] - | 1| 2600|2.5644e-05| 0.5| 12.353| 0.015625| [ INFO : 2022-12-03 03:09:33,149 ] - | 1| 2700|2.6404e-05| 0.5| 12.352| 0.015625| [ INFO : 2022-12-03 03:10:10,267 ] - | 1| 2800| 2.715e-05| 0.5| 12.352| 0.016183| [ INFO : 2022-12-03 03:10:47,349 ] - | 1| 2900| 2.788e-05| 0.5| 12.348| 0.016164| [ INFO : 2022-12-03 03:11:24,459 ] - | 1| 3000|2.8597e-05| 0.5| 12.342| 0.016146| [ INFO : 2022-12-03 03:12:01,693 ] - | 1| 3100|2.9299e-05| 0.5| 12.336| 0.015625| [ INFO : 2022-12-03 03:12:38,796 ] - | 1| 3200|2.9988e-05| 0.5| 12.332| 0.015137| [ INFO : 2022-12-03 03:13:15,911 ] - | 1| 3300|3.0662e-05| 0.5| 12.33| 0.015152| [ INFO : 2022-12-03 03:13:53,012 ] - | 1| 3400|3.1323e-05| 0.5| 12.326| 0.015625| [ INFO : 2022-12-03 03:14:30,112 ] - | 1| 3500| 3.197e-05| 0.5| 12.327| 0.016518| [ INFO : 2022-12-03 03:15:07,216 ] - | 1| 3600|3.2604e-05| 0.5| 12.321| 0.016059| [ INFO : 2022-12-03 03:15:44,466 ] - | 1| 3700|3.3225e-05| 0.5| 12.319| 0.016047| [ INFO : 2022-12-03 03:16:21,569 ] - | 1| 3800|3.3833e-05| 0.5| 12.315| 0.016859| [ INFO : 2022-12-03 03:16:58,648 ] - | 1| 3900|3.4429e-05| 0.5| 12.312| 0.017228| [ INFO : 2022-12-03 03:17:35,742 ] - | 1| 4000|3.5012e-05| 0.5| 12.311| 0.016797| [ INFO : 2022-12-03 03:18:12,845 ] - | 1| 4100|3.5582e-05| 0.5| 12.306| 0.016768| [ INFO : 2022-12-03 03:18:49,952 ] - | 1| 4200| 3.614e-05| 0.5| 12.301| 0.017113| [ INFO : 2022-12-03 03:19:27,211 ] - | 1| 4300|3.6686e-05| 0.5| 12.297| 0.017805| [ INFO : 2022-12-03 03:20:04,312 ] - | 1| 4400| 3.722e-05| 0.5| 12.291| 0.018821| [ INFO : 2022-12-03 03:20:41,407 ] - | 1| 4500|3.7743e-05| 0.5| 12.287| 0.019097| [ INFO : 2022-12-03 03:21:18,501 ] - | 1| 4600|3.8254e-05| 0.5| 12.285| 0.019701| [ INFO : 2022-12-03 03:21:55,597 ] - | 1| 4700|3.8753e-05| 0.5| 12.282| 0.019614| [ INFO : 2022-12-03 03:22:32,717 ] - | 1| 4800|3.9242e-05| 0.5| 12.278| 0.019857| [ INFO : 2022-12-03 03:23:09,987 ] - | 1| 4900|3.9719e-05| 0.5| 12.275| 0.020408| [ INFO : 2022-12-03 03:23:47,085 ] - | 1| 5000|4.0185e-05| 0.5| 12.272| 0.021563| [ INFO : 2022-12-03 03:24:24,188 ] - | 1| 5100| 4.064e-05| 0.5| 12.267| 0.022978| [ INFO : 2022-12-03 03:25:01,271 ] - | 1| 5200|4.1085e-05| 0.5| 12.262| 0.022837| [ INFO : 2022-12-03 03:25:38,378 ] - | 1| 5300|4.1519e-05| 0.5| 12.257| 0.023585| [ INFO : 2022-12-03 03:26:15,468 ] - | 1| 5400|4.1943e-05| 0.5| 12.255| 0.025463| [ INFO : 2022-12-03 03:26:52,612 ] - | 1| 5500|4.2357e-05| 0.5| 12.251| 0.026136| [ INFO : 2022-12-03 03:27:29,807 ] - | 1| 5600| 4.276e-05| 0.5| 12.246| 0.026507| [ INFO : 2022-12-03 03:28:06,919 ] - | 1| 5700|4.3154e-05| 0.5| 12.243| 0.02659| [ INFO : 2022-12-03 03:28:44,029 ] - | 1| 5800|4.3538e-05| 0.5| 12.239| 0.02694| [ INFO : 2022-12-03 03:29:21,138 ] - | 1| 5900|4.3912e-05| 0.5| 12.236| 0.027542| [ INFO : 2022-12-03 03:29:58,253 ] - | 1| 6000|4.4277e-05| 0.5| 12.235| 0.028385| [ INFO : 2022-12-03 03:30:35,345 ] - | 1| 6100|4.4632e-05| 0.5| 12.232| 0.028689| [ INFO : 2022-12-03 03:31:12,565 ] - | 1| 6200|4.4978e-05| 0.5| 12.229| 0.02873| [ INFO : 2022-12-03 03:31:49,658 ] - | 1| 6300|4.5315e-05| 0.5| 12.226| 0.029514| [ INFO : 2022-12-03 03:32:26,746 ] - | 1| 6400|4.5643e-05| 0.5| 12.223| 0.030029| [ INFO : 2022-12-03 03:33:03,843 ] - | 1| 6500|4.5962e-05| 0.5| 12.219| 0.03125| [ INFO : 2022-12-03 03:33:40,948 ] - | 1| 6600|4.6273e-05| 0.5| 12.217| 0.031723| [ INFO : 2022-12-03 03:34:18,056 ] - | 1| 6700|4.6575e-05| 0.5| 12.214| 0.033116| [ INFO : 2022-12-03 03:34:55,296 ] - | 1| 6800|4.6868e-05| 0.5| 12.21| 0.034697| [ INFO : 2022-12-03 03:35:32,409 ] - | 1| 6900|4.7153e-05| 0.5| 12.207| 0.034873| [ INFO : 2022-12-03 03:36:09,538 ] - | 1| 7000| 4.743e-05| 0.5| 12.203| 0.035938| [ INFO : 2022-12-03 03:36:46,645 ] - | 1| 7100|4.7698e-05| 0.5| 12.199| 0.037852| [ INFO : 2022-12-03 03:37:23,755 ] - | 1| 7200|4.7959e-05| 0.5| 12.195| 0.039497| [ INFO : 2022-12-03 03:38:00,885 ] - | 1| 7300|4.8212e-05| 0.5| 12.193| 0.040668| [ INFO : 2022-12-03 03:38:38,148 ] - | 1| 7400|4.8457e-05| 0.5| 12.19| 0.042441| [ INFO : 2022-12-03 03:39:15,257 ] - | 1| 7500|4.8694e-05| 0.5| 12.188| 0.042708| [ INFO : 2022-12-03 03:39:52,363 ] - | 1| 7600|4.8924e-05| 0.5| 12.186| 0.043791| [ INFO : 2022-12-03 03:40:29,467 ] - | 1| 7700|4.9146e-05| 0.5| 12.183| 0.046266| [ INFO : 2022-12-03 03:41:06,576 ] - | 1| 7800|4.9362e-05| 0.5| 12.182| 0.047276| [ INFO : 2022-12-03 03:41:43,664 ] - | 1| 7900|4.9569e-05| 0.5| 12.178| 0.048655| [ INFO : 2022-12-03 03:42:20,748 ] - | 1| 8000| 4.977e-05| 0.5| 12.175| 0.049805| [ INFO : 2022-12-03 03:42:57,815 ] - | 1| 8100|4.9964e-05| 0.5| 12.172| 0.050347| [ INFO : 2022-12-03 03:43:02,511 ] - | 1| 8112|4.9987e-05| 0.5| 12.172| 0.050465| [ INFO : 2022-12-03 03:43:46,900 ] - | 2| 100|4.9579e-05| 0.5| 11.93| 0.20313| [ INFO : 2022-12-03 03:44:23,982 ] - | 2| 200|4.9158e-05| 0.5| 11.958| 0.16406| [ INFO : 2022-12-03 03:45:01,078 ] - | 2| 300| 4.874e-05| 0.5| 11.965| 0.14063| [ INFO : 2022-12-03 03:45:38,172 ] - | 2| 400|4.8325e-05| 0.5| 11.938| 0.14453| [ INFO : 2022-12-03 03:46:15,273 ] - | 2| 500|4.7914e-05| 0.5| 11.939| 0.13437| [ INFO : 2022-12-03 03:46:52,511 ] - | 2| 600|4.7507e-05| 0.5| 11.943| 0.13542| [ INFO : 2022-12-03 03:47:29,601 ] - | 2| 700|4.7103e-05| 0.5| 11.95| 0.13393| [ INFO : 2022-12-03 03:48:06,693 ] - | 2| 800|4.6703e-05| 0.5| 11.944| 0.13672| [ INFO : 2022-12-03 03:48:43,790 ] - | 2| 900|4.6306e-05| 0.5| 11.942| 0.13715| [ INFO : 2022-12-03 03:49:20,886 ] - | 2| 1000|4.5912e-05| 0.5| 11.946| 0.15| [ INFO : 2022-12-03 03:49:57,982 ] - | 2| 1100|4.5522e-05| 0.5| 11.948| 0.15767| [ INFO : 2022-12-03 03:50:35,232 ] - | 2| 1200|4.5135e-05| 0.5| 11.942| 0.16536| [ INFO : 2022-12-03 03:51:12,333 ] - | 2| 1300|4.4751e-05| 0.5| 11.95| 0.16707| [ INFO : 2022-12-03 03:51:49,403 ] - | 2| 1400| 4.437e-05| 0.5| 11.936| 0.16741| [ INFO : 2022-12-03 03:52:26,514 ] - | 2| 1500|4.3993e-05| 0.5| 11.931| 0.16667| [ INFO : 2022-12-03 03:53:03,598 ] - | 2| 1600|4.3619e-05| 0.5| 11.922| 0.16895| [ INFO : 2022-12-03 03:53:40,699 ] - | 2| 1700|4.3248e-05| 0.5| 11.919| 0.17096| [ INFO : 2022-12-03 03:54:17,972 ] - | 2| 1800|4.2881e-05| 0.5| 11.915| 0.17187| [ INFO : 2022-12-03 03:54:55,091 ] - | 2| 1900|4.2516e-05| 0.5| 11.91| 0.17516| [ INFO : 2022-12-03 03:55:32,188 ] - | 2| 2000|4.2154e-05| 0.5| 11.905| 0.18047| [ INFO : 2022-12-03 03:56:09,303 ] - | 2| 2100|4.1796e-05| 0.5| 11.901| 0.18452| [ INFO : 2022-12-03 03:56:46,397 ] - | 2| 2200|4.1441e-05| 0.5| 11.9| 0.18608| [ INFO : 2022-12-03 03:57:23,492 ] - | 2| 2300|4.1088e-05| 0.5| 11.898| 0.18614| [ INFO : 2022-12-03 03:58:00,735 ] - | 2| 2400|4.0739e-05| 0.5| 11.897| 0.18815| [ INFO : 2022-12-03 03:58:37,843 ] - | 2| 2500|4.0393e-05| 0.5| 11.894| 0.18812| [ INFO : 2022-12-03 03:59:14,929 ] - | 2| 2600|4.0049e-05| 0.5| 11.893| 0.19171| [ INFO : 2022-12-03 03:59:52,018 ] - | 2| 2700|3.9709e-05| 0.5| 11.89| 0.19618| [ INFO : 2022-12-03 04:00:29,109 ] - | 2| 2800|3.9371e-05| 0.5| 11.891| 0.19866| [ INFO : 2022-12-03 04:01:06,206 ] - | 2| 2900|3.9036e-05| 0.5| 11.888| 0.2069| [ INFO : 2022-12-03 04:01:43,311 ] - | 2| 3000|3.8705e-05| 0.5| 11.883| 0.20938| [ INFO : 2022-12-03 04:02:20,500 ] - | 2| 3100|3.8375e-05| 0.5| 11.881| 0.21069| [ INFO : 2022-12-03 04:02:57,586 ] - | 2| 3200|3.8049e-05| 0.5| 11.88| 0.21436| [ INFO : 2022-12-03 04:03:34,687 ] - | 2| 3300|3.7726e-05| 0.5| 11.879| 0.21638| [ INFO : 2022-12-03 04:04:11,792 ] - | 2| 3400|3.7405e-05| 0.5| 11.877| 0.22013| [ INFO : 2022-12-03 04:04:48,852 ] - | 2| 3500|3.7087e-05| 0.5| 11.871| 0.22411| [ INFO : 2022-12-03 04:05:25,944 ] - | 2| 3600|3.6772e-05| 0.5| 11.868| 0.227| [ INFO : 2022-12-03 04:06:03,174 ] - | 2| 3700|3.6459e-05| 0.5| 11.867| 0.22424| [ INFO : 2022-12-03 04:06:40,282 ] - | 2| 3800|3.6149e-05| 0.5| 11.864| 0.22615| [ INFO : 2022-12-03 04:07:17,378 ] - | 2| 3900|3.5842e-05| 0.5| 11.862| 0.23037| [ INFO : 2022-12-03 04:07:54,466 ] - | 2| 4000|3.5537e-05| 0.5| 11.861| 0.23164| [ INFO : 2022-12-03 04:08:31,575 ] - | 2| 4100|3.5235e-05| 0.5| 11.859| 0.23552| [ INFO : 2022-12-03 04:09:08,669 ] - | 2| 4200|3.4935e-05| 0.5| 11.857| 0.23847| [ INFO : 2022-12-03 04:09:45,905 ] - | 2| 4300|3.4638e-05| 0.5| 11.856| 0.24237| [ INFO : 2022-12-03 04:10:22,981 ] - | 2| 4400|3.4344e-05| 0.5| 11.853| 0.24219| [ INFO : 2022-12-03 04:11:00,074 ] - | 2| 4500|3.4052e-05| 0.5| 11.851| 0.24514| [ INFO : 2022-12-03 04:11:37,159 ] - | 2| 4600|3.3762e-05| 0.5| 11.849| 0.2449| [ INFO : 2022-12-03 04:12:14,246 ] - | 2| 4700|3.3475e-05| 0.5| 11.848| 0.25033| [ INFO : 2022-12-03 04:12:51,320 ] - | 2| 4800|3.3191e-05| 0.5| 11.845| 0.25423| [ INFO : 2022-12-03 04:13:28,522 ] - | 2| 4900|3.2908e-05| 0.5| 11.844| 0.25606| [ INFO : 2022-12-03 04:14:05,625 ] - | 2| 5000|3.2629e-05| 0.5| 11.841| 0.25906| [ INFO : 2022-12-03 04:14:42,711 ] - | 2| 5100|3.2351e-05| 0.5| 11.838| 0.26011| [ INFO : 2022-12-03 04:15:19,811 ] - | 2| 5200|3.2076e-05| 0.5| 11.836| 0.26202| [ INFO : 2022-12-03 04:15:56,910 ] - | 2| 5300|3.1803e-05| 0.5| 11.834| 0.26297| [ INFO : 2022-12-03 04:16:33,999 ] - | 2| 5400|3.1533e-05| 0.5| 11.833| 0.2662| [ INFO : 2022-12-03 04:17:11,129 ] - | 2| 5500|3.1265e-05| 0.5| 11.831| 0.27017| [ INFO : 2022-12-03 04:17:48,315 ] - | 2| 5600|3.0999e-05| 0.5| 11.829| 0.26981| [ INFO : 2022-12-03 04:18:25,391 ] - | 2| 5700|3.0736e-05| 0.5| 11.825| 0.27275| [ INFO : 2022-12-03 04:19:02,482 ] - | 2| 5800|3.0474e-05| 0.5| 11.824| 0.27559| [ INFO : 2022-12-03 04:19:39,565 ] - | 2| 5900|3.0215e-05| 0.5| 11.821| 0.27807| [ INFO : 2022-12-03 04:20:16,670 ] - | 2| 6000|2.9958e-05| 0.5| 11.82| 0.28021| [ INFO : 2022-12-03 04:20:53,762 ] - | 2| 6100|2.9704e-05| 0.5| 11.818| 0.28304| [ INFO : 2022-12-03 04:21:30,989 ] - | 2| 6200|2.9451e-05| 0.5| 11.817| 0.28755| [ INFO : 2022-12-03 04:22:08,065 ] - | 2| 6300|2.9201e-05| 0.5| 11.816| 0.28919| [ INFO : 2022-12-03 04:22:45,141 ] - | 2| 6400|2.8952e-05| 0.5| 11.814| 0.29272| [ INFO : 2022-12-03 04:23:22,242 ] - | 2| 6500|2.8706e-05| 0.5| 11.81| 0.29495| [ INFO : 2022-12-03 04:23:59,334 ] - | 2| 6600|2.8462e-05| 0.5| 11.808| 0.2964| [ INFO : 2022-12-03 04:24:36,442 ] - | 2| 6700| 2.822e-05| 0.5| 11.808| 0.29921| [ INFO : 2022-12-03 04:25:13,692 ] - | 2| 6800| 2.798e-05| 0.5| 11.806| 0.30193| [ INFO : 2022-12-03 04:25:50,779 ] - | 2| 6900|2.7742e-05| 0.5| 11.806| 0.3048| [ INFO : 2022-12-03 04:26:27,873 ] - | 2| 7000|2.7506e-05| 0.5| 11.803| 0.30826| [ INFO : 2022-12-03 04:27:04,951 ] - | 2| 7100|2.7273e-05| 0.5| 11.801| 0.30986| [ INFO : 2022-12-03 04:27:42,051 ] - | 2| 7200|2.7041e-05| 0.5| 11.798| 0.31141| [ INFO : 2022-12-03 04:28:19,145 ] - | 2| 7300|2.6811e-05| 0.5| 11.795| 0.31507| [ INFO : 2022-12-03 04:28:56,375 ] - | 2| 7400|2.6583e-05| 0.5| 11.792| 0.3163| [ INFO : 2022-12-03 04:29:33,490 ] - | 2| 7500|2.6357e-05| 0.5| 11.789| 0.31875| [ INFO : 2022-12-03 04:30:10,591 ] - | 2| 7600|2.6133e-05| 0.5| 11.788| 0.32175| [ INFO : 2022-12-03 04:30:47,701 ] - | 2| 7700|2.5911e-05| 0.5| 11.788| 0.32447| [ INFO : 2022-12-03 04:31:24,804 ] - | 2| 7800| 2.569e-05| 0.5| 11.787| 0.32552| [ INFO : 2022-12-03 04:32:01,908 ] - | 2| 7900|2.5472e-05| 0.5| 11.785| 0.32911| [ INFO : 2022-12-03 04:32:38,985 ] - | 2| 8000|2.5255e-05| 0.5| 11.784| 0.33086| [ INFO : 2022-12-03 04:33:16,037 ] - | 2| 8100|2.5041e-05| 0.5| 11.782| 0.33468| [ INFO : 2022-12-03 04:33:20,812 ] - | 2| 8112|2.5015e-05| 0.5| 11.782| 0.33496| [ INFO : 2022-12-03 04:33:20,897 ] - +----------+----------+----------+----------+----------+----------+ Namespace(dst_model='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/avg_model.pt', src_path='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models', num=1, min_epoch=0, max_epoch=65536) ['exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/model_2.pt'] Processing exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/model_2.pt Saving to exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/avg_model.pt Extract embeddings ... extract_embedding from /home/yoos/Documents/data/cnceleb_train/raw.list, wavs_num: 519590 extract_embedding from /home/yoos/Documents/data/eval/raw.list, wavs_num: 18772 Fail eval Success cnceleb_train Embedding dir is (exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/embeddings). mean vector of enroll 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [00:00<00:00, 86489.59it/s] Score ... apply cosine scoring ... CNC-Eval-Concat.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Concat.lst: 0%| | 4884/3484292 [00:00<03:01, 19207.42it/s] Traceback (most recent call last): File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 96, in fire.Fire(main) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, kwargs) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 92, in main trials_cosine_score(eval_scp_path, store_score_dir, mean_vec_path, trials) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 62, in trials_cosine_score emb1, emb2 = emb_dict[segs[0]], emb_dict[segs[1]] KeyError: 'test/id00856-speech-01-001.wav' CNC-Eval-Avg.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Avg.lst: 0%| | 4884/3484292 [00:00<03:02, 19032.16it/s] Traceback (most recent call last): File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 96, in fire.Fire(main) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 92, in main trials_cosine_score(eval_scp_path, store_score_dir, mean_vec_path, trials) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 62, in trials_cosine_score emb1, emb2 = emb_dict[segs[0]], emb_dict[segs[1]] KeyError: 'test/id00856-speech-01-001.wav' compute metrics (EER/minDCF) ... ---- CNC-Eval-Concat.lst.score ----- EER = 20.000 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.507 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/scores/CNC-Eval-Concat.lst.score.det.png ---- CNC-Eval-Avg.lst.score ----- EER = 16.923 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.528 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/scores/CNC-Eval-Avg.lst.score.det.png Score norm ... compute mean xvector 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2793/2793 [00:00<00:00, 10002.55it/s] compute norm score 2022-12-03 04:51:56,787 INFO get embedding ... 2022-12-03 04:51:56,919 INFO computing normed score ... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4884/4884 [00:00<00:00, 932322.08it/s] 2022-12-03 04:51:57,523 INFO Over! 2022-12-03 04:51:57,694 INFO get embedding ... 2022-12-03 04:51:57,817 INFO computing normed score ... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4884/4884 [00:00<00:00, 860388.12it/s] 2022-12-03 04:51:58,478 INFO Over! compute metrics ---- cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score ----- EER = 20.000 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.507 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/scores/cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score.det.png ---- cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score ----- EER = 16.410 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.502 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/scores/cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score.det.png Export the best model ... ResNet( (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (layer1): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (1): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer2): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer3): Sequential( (0): BasicBlock( (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (4): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (5): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer4): Sequential( (0): BasicBlock( (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (pool): TSTP() (seg_1): Linear(in_features=5120, out_features=256, bias=True) (seg_bn_1): Identity() (seg_2): Identity() ) Export model successfully, see exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch6-LM/models/final.zip (wespeaker) yoos@yoos-System-Product-Name:~/Documents/code/wespeaker/examples/cnceleb/v2$ bash run.sh Preparing datasets ... combine short audios and convert flac to wav ... combine audios for cnceleb2 Calcualte the duration for each audio file, this may take for a while... Calcualte the duration done! 100%|█████████████████████████████████████████████████████| 461307/461307 [04:02<00:00, 1902.92it/s] combine audios for cnceleb1_dev Calcualte the duration for each audio file, this may take for a while... Calcualte the duration done! 100%|████████████████████████████████████████████████████████| 58283/58283 [01:07<00:00, 860.51it/s] combine audios for cnceleb1_eval 100%|███████████████████████████████████████████████████████| 18579/18579 [00:14<00:00, 1283.71it/s] combine audios for cnceleb1_enroll 100%|███████████████████████████████████████████████████████| 17973/17973 [00:14<00:00, 1199.72it/s] convert success Prepare wav.scp for each dataset ... Prepare train data including CN-Celeb_wav/dev and CN-Celeb2_wav ... Prepare data for testing ... Prepare data for enroll ... Prepare evalution trials ... Success !!! Now data preparation is done !!! Covert train and test data to raw... (wespeaker) yoos@yoos-System-Product-Name:~/Documents/code/wespeaker/examples/cnceleb/v2$ bash run.sh Start training ... [ INFO : 2022-12-03 11:16:31,605 ] - exp_dir is: exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3 [ INFO : 2022-12-03 11:16:31,605 ] - <== Passed Arguments ==> [ INFO : 2022-12-03 11:16:31,605 ] - {'data_type': 'raw', [ INFO : 2022-12-03 11:16:31,605 ] - 'dataloader_args': {'batch_size': 256, [ INFO : 2022-12-03 11:16:31,605 ] - 'drop_last': True, [ INFO : 2022-12-03 11:16:31,605 ] - 'num_workers': 16, [ INFO : 2022-12-03 11:16:31,605 ] - 'pin_memory': False, [ INFO : 2022-12-03 11:16:31,606 ] - 'prefetch_factor': 8}, [ INFO : 2022-12-03 11:16:31,606 ] - 'dataset_args': {'aug_prob': 0.6, [ INFO : 2022-12-03 11:16:31,606 ] - 'fbank_args': {'dither': 1.0, [ INFO : 2022-12-03 11:16:31,606 ] - 'frame_length': 25, [ INFO : 2022-12-03 11:16:31,606 ] - 'frame_shift': 10, [ INFO : 2022-12-03 11:16:31,606 ] - 'num_mel_bins': 80}, [ INFO : 2022-12-03 11:16:31,606 ] - 'num_frms': 200, [ INFO : 2022-12-03 11:16:31,606 ] - 'resample_rate': 16000, [ INFO : 2022-12-03 11:16:31,606 ] - 'shuffle': True, [ INFO : 2022-12-03 11:16:31,606 ] - 'shuffle_args': {'shuffle_size': 2500}, [ INFO : 2022-12-03 11:16:31,606 ] - 'spec_aug': False, [ INFO : 2022-12-03 11:16:31,606 ] - 'spec_aug_args': {'max_f': 8, [ INFO : 2022-12-03 11:16:31,606 ] - 'max_t': 10, [ INFO : 2022-12-03 11:16:31,606 ] - 'num_f_mask': 1, [ INFO : 2022-12-03 11:16:31,606 ] - 'num_t_mask': 1, [ INFO : 2022-12-03 11:16:31,606 ] - 'prob': 0.6}, [ INFO : 2022-12-03 11:16:31,606 ] - 'speed_perturb': True}, [ INFO : 2022-12-03 11:16:31,606 ] - 'enable_amp': False, [ INFO : 2022-12-03 11:16:31,606 ] - 'exp_dir': 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3', [ INFO : 2022-12-03 11:16:31,606 ] - 'gpus': [0], [ INFO : 2022-12-03 11:16:31,606 ] - 'log_batch_interval': 100, [ INFO : 2022-12-03 11:16:31,606 ] - 'loss': 'CrossEntropyLoss', [ INFO : 2022-12-03 11:16:31,606 ] - 'loss_args': {}, [ INFO : 2022-12-03 11:16:31,606 ] - 'margin_scheduler': 'MarginScheduler', [ INFO : 2022-12-03 11:16:31,606 ] - 'margin_update': {'final_margin': 0.2, [ INFO : 2022-12-03 11:16:31,606 ] - 'fix_start_epoch': 40, [ INFO : 2022-12-03 11:16:31,606 ] - 'increase_start_epoch': 20, [ INFO : 2022-12-03 11:16:31,606 ] - 'increase_type': 'exp', [ INFO : 2022-12-03 11:16:31,606 ] - 'initial_margin': 0.0, [ INFO : 2022-12-03 11:16:31,606 ] - 'update_margin': True}, [ INFO : 2022-12-03 11:16:31,606 ] - 'model': 'ResNet34', [ INFO : 2022-12-03 11:16:31,606 ] - 'model_args': {'embed_dim': 256, [ INFO : 2022-12-03 11:16:31,606 ] - 'feat_dim': 80, [ INFO : 2022-12-03 11:16:31,606 ] - 'pooling_func': 'TSTP', [ INFO : 2022-12-03 11:16:31,606 ] - 'two_emb_layer': False}, [ INFO : 2022-12-03 11:16:31,606 ] - 'model_init': None, [ INFO : 2022-12-03 11:16:31,606 ] - 'noise_data': '/home/yoos/Documents/data/musan/lmdb', [ INFO : 2022-12-03 11:16:31,606 ] - 'num_avg': 2, [ INFO : 2022-12-03 11:16:31,606 ] - 'num_epochs': 3, [ INFO : 2022-12-03 11:16:31,606 ] - 'optimizer': 'SGD', [ INFO : 2022-12-03 11:16:31,606 ] - 'optimizer_args': {'momentum': 0.9, 'nesterov': True, 'weight_decay': 0.0001}, [ INFO : 2022-12-03 11:16:31,606 ] - 'projection_args': {'easy_margin': False, [ INFO : 2022-12-03 11:16:31,606 ] - 'project_type': 'arc_margin', [ INFO : 2022-12-03 11:16:31,606 ] - 'scale': 32.0}, [ INFO : 2022-12-03 11:16:31,606 ] - 'reverb_data': '/home/yoos/Documents/data/rirs/lmdb', [ INFO : 2022-12-03 11:16:31,606 ] - 'save_epoch_interval': 2, [ INFO : 2022-12-03 11:16:31,606 ] - 'scheduler': 'ExponentialDecrease', [ INFO : 2022-12-03 11:16:31,606 ] - 'scheduler_args': {'final_lr': 5e-05, [ INFO : 2022-12-03 11:16:31,606 ] - 'initial_lr': 0.1, [ INFO : 2022-12-03 11:16:31,606 ] - 'warm_from_zero': True, [ INFO : 2022-12-03 11:16:31,606 ] - 'warm_up_epoch': 6}, [ INFO : 2022-12-03 11:16:31,606 ] - 'seed': 42, [ INFO : 2022-12-03 11:16:31,606 ] - 'train_data': '/home/yoos/Documents/data/cnceleb_train/raw.list', [ INFO : 2022-12-03 11:16:31,606 ] - 'train_label': '/home/yoos/Documents/data/cnceleb_train/utt2spk'} [ INFO : 2022-12-03 11:16:32,079 ] - <== Data statistics ==> [ INFO : 2022-12-03 11:16:32,079 ] - train data num: 519590, spk num: 2793 [ INFO : 2022-12-03 11:16:32,157 ] - <== Dataloaders ==> [ INFO : 2022-12-03 11:16:32,157 ] - train dataloaders created [ INFO : 2022-12-03 11:16:32,157 ] - loader size: 2029 [ INFO : 2022-12-03 11:16:32,157 ] - <== Model ==> [ INFO : 2022-12-03 11:16:32,206 ] - speaker_model size: 6634336 [ INFO : 2022-12-03 11:16:32,206 ] - Train model from scratch ... [ INFO : 2022-12-03 11:16:32,213 ] - ResNet( [ INFO : 2022-12-03 11:16:32,213 ] - (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (layer1): Sequential( [ INFO : 2022-12-03 11:16:32,213 ] - (0): BasicBlock( [ INFO : 2022-12-03 11:16:32,213 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,213 ] - ) [ INFO : 2022-12-03 11:16:32,213 ] - (1): BasicBlock( [ INFO : 2022-12-03 11:16:32,213 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,213 ] - ) [ INFO : 2022-12-03 11:16:32,213 ] - (2): BasicBlock( [ INFO : 2022-12-03 11:16:32,213 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,213 ] - ) [ INFO : 2022-12-03 11:16:32,213 ] - ) [ INFO : 2022-12-03 11:16:32,213 ] - (layer2): Sequential( [ INFO : 2022-12-03 11:16:32,213 ] - (0): BasicBlock( [ INFO : 2022-12-03 11:16:32,213 ] - (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - (shortcut): Sequential( [ INFO : 2022-12-03 11:16:32,213 ] - (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,213 ] - ) [ INFO : 2022-12-03 11:16:32,213 ] - ) [ INFO : 2022-12-03 11:16:32,213 ] - (1): BasicBlock( [ INFO : 2022-12-03 11:16:32,213 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,213 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (2): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (3): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (layer3): Sequential( [ INFO : 2022-12-03 11:16:32,214 ] - (0): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential( [ INFO : 2022-12-03 11:16:32,214 ] - (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (1): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (2): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (3): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (4): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (5): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (layer4): Sequential( [ INFO : 2022-12-03 11:16:32,214 ] - (0): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential( [ INFO : 2022-12-03 11:16:32,214 ] - (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (1): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (2): BasicBlock( [ INFO : 2022-12-03 11:16:32,214 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 11:16:32,214 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 11:16:32,214 ] - (shortcut): Sequential() [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - ) [ INFO : 2022-12-03 11:16:32,214 ] - (pool): TSTP() [ INFO : 2022-12-03 11:16:32,214 ] - (seg_1): Linear(in_features=5120, out_features=256, bias=True) [ INFO : 2022-12-03 11:16:32,214 ] - (seg_bn_1): Identity() [ INFO : 2022-12-03 11:16:32,215 ] - (seg_2): Identity() [ INFO : 2022-12-03 11:16:32,215 ] - (projection): ArcMarginProduct( [ INFO : 2022-12-03 11:16:32,215 ] - in_features=256, out_features=8379, scale=32.0, [ INFO : 2022-12-03 11:16:32,215 ] - margin=0.0, easy_margin=False [ INFO : 2022-12-03 11:16:32,215 ] - ) [ INFO : 2022-12-03 11:16:32,215 ] - ) [ INFO : 2022-12-03 11:16:32,399 ] - start_epoch: 1 [ INFO : 2022-12-03 11:16:32,413 ] - <== Loss ==> [ INFO : 2022-12-03 11:16:32,413 ] - loss criterion is: CrossEntropyLoss [ INFO : 2022-12-03 11:16:32,413 ] - <== Optimizer ==> [ INFO : 2022-12-03 11:16:32,413 ] - optimizer is: SGD [ INFO : 2022-12-03 11:16:32,413 ] - <== Scheduler ==> [ INFO : 2022-12-03 11:16:32,413 ] - scheduler is: ExponentialDecrease [ INFO : 2022-12-03 11:16:32,413 ] - <== MarginScheduler ==> [ INFO : 2022-12-03 11:16:32,415 ] - <========== Training process ==========> [ INFO : 2022-12-03 11:16:32,415 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 11:16:32,415 ] - | Epoch| Batch| Lr| Margin| Loss| Acc| [ INFO : 2022-12-03 11:16:32,415 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 11:16:51,808 ] - Reducer buckets have been rebuilt in this iteration. [ INFO : 2022-12-03 11:17:43,833 ] - | 1| 100| 0.0028746| 0| 9.5133| 0.14063| [ INFO : 2022-12-03 11:18:36,449 ] - | 1| 200| 0.0050999| 0| 9.3101| 0.16406| [ INFO : 2022-12-03 11:19:27,513 ] - | 1| 300| 0.0067631| 0| 9.2605| 0.1849| [ INFO : 2022-12-03 11:20:20,001 ] - | 1| 400| 0.0079656| 0| 9.166| 0.25195| [ INFO : 2022-12-03 11:21:12,292 ] - | 1| 500| 0.0087926| 0| 9.0315| 0.40234| [ INFO : 2022-12-03 11:22:05,094 ] - | 1| 600| 0.0093156| 0| 8.8599| 0.62891| [ INFO : 2022-12-03 11:22:57,795 ] - | 1| 700| 0.0095947| 0| 8.662| 0.89844| [ INFO : 2022-12-03 11:23:49,258 ] - | 1| 800| 0.0096799| 0| 8.4577| 1.3359| [ INFO : 2022-12-03 11:24:40,591 ] - | 1| 900| 0.0096128| 0| 8.2498| 1.8984| [ INFO : 2022-12-03 11:25:31,815 ] - | 1| 1000| 0.0094281| 0| 8.0367| 2.6164| [ INFO : 2022-12-03 11:26:22,866 ] - | 1| 1100| 0.0091544| 0| 7.8263| 3.4858| [ INFO : 2022-12-03 11:27:14,152 ] - | 1| 1200| 0.0088149| 0| 7.6217| 4.4717| [ INFO : 2022-12-03 11:28:05,433 ] - | 1| 1300| 0.008429| 0| 7.4245| 5.5009| [ INFO : 2022-12-03 11:28:56,500 ] - | 1| 1400| 0.0080123| 0| 7.2343| 6.6401| [ INFO : 2022-12-03 11:29:47,828 ] - | 1| 1500| 0.0075772| 0| 7.0571| 7.7867| [ INFO : 2022-12-03 11:30:38,960 ] - | 1| 1600| 0.0071339| 0| 6.8854| 8.9854| [ INFO : 2022-12-03 11:31:30,248 ] - | 1| 1700| 0.0066902| 0| 6.7232| 10.186| [ INFO : 2022-12-03 11:32:21,572 ] - | 1| 1800| 0.0062524| 0| 6.5707| 11.365| [ INFO : 2022-12-03 11:33:12,579 ] - | 1| 1900| 0.0058252| 0| 6.4269| 12.502| [ INFO : 2022-12-03 11:34:03,064 ] - | 1| 2000| 0.0054121| 0| 6.2918| 13.631| [ INFO : 2022-12-03 11:34:11,424 ] - | 1| 2016| 0.0053475| 0| 6.2705| 13.82| [ INFO : 2022-12-03 11:35:15,182 ] - | 2| 100| 0.0049042| 0| 3.5367| 37.559| [ INFO : 2022-12-03 11:36:08,801 ] - | 2| 200| 0.0045319| 0| 3.5052| 38.029| [ INFO : 2022-12-03 11:37:00,316 ] - | 2| 300| 0.0041794| 0| 3.4621| 38.741| [ INFO : 2022-12-03 11:37:51,856 ] - | 2| 400| 0.0038473| 0| 3.417| 39.491| [ INFO : 2022-12-03 11:38:43,197 ] - | 2| 500| 0.0035355| 0| 3.3761| 40.209| [ INFO : 2022-12-03 11:39:34,814 ] - | 2| 600| 0.0032439| 0| 3.3377| 40.85| [ INFO : 2022-12-03 11:40:26,513 ] - | 2| 700| 0.002972| 0| 3.3053| 41.334| [ INFO : 2022-12-03 11:41:17,795 ] - | 2| 800| 0.0027193| 0| 3.2692| 41.912| [ INFO : 2022-12-03 11:42:09,497 ] - | 2| 900| 0.002485| 0| 3.2346| 42.459| [ INFO : 2022-12-03 11:43:01,152 ] - | 2| 1000| 0.0022682| 0| 3.207| 42.947| [ INFO : 2022-12-03 11:43:52,524 ] - | 2| 1100| 0.002068| 0| 3.1782| 43.417| [ INFO : 2022-12-03 11:44:44,081 ] - | 2| 1200| 0.0018836| 0| 3.1513| 43.861| [ INFO : 2022-12-03 11:45:35,673 ] - | 2| 1300| 0.001714| 0| 3.1273| 44.27| [ INFO : 2022-12-03 11:46:27,131 ] - | 2| 1400| 0.0015582| 0| 3.1026| 44.706| [ INFO : 2022-12-03 11:47:18,682 ] - | 2| 1500| 0.0014154| 0| 3.0805| 45.08| [ INFO : 2022-12-03 11:48:09,791 ] - | 2| 1600| 0.0012847| 0| 3.0588| 45.444| [ INFO : 2022-12-03 11:49:00,986 ] - | 2| 1700| 0.0011651| 0| 3.0386| 45.789| [ INFO : 2022-12-03 11:49:52,313 ] - | 2| 1800| 0.0010559| 0| 3.0181| 46.139| [ INFO : 2022-12-03 11:50:43,376 ] - | 2| 1900|0.00095634| 0| 2.9983| 46.479| [ INFO : 2022-12-03 11:51:33,865 ] - | 2| 2000|0.00086556| 0| 2.9814| 46.763| [ INFO : 2022-12-03 11:51:43,015 ] - | 2| 2016|0.00085181| 0| 2.9788| 46.797| [ INFO : 2022-12-03 11:52:47,517 ] - | 3| 100|0.00076038| 0| 2.6033| 53.141| [ INFO : 2022-12-03 11:53:39,958 ] - | 3| 200|0.00068727| 0| 2.5967| 53.172| [ INFO : 2022-12-03 11:54:31,176 ] - | 3| 300|0.00062084| 0| 2.5937| 53.19| [ INFO : 2022-12-03 11:55:22,486 ] - | 3| 400|0.00056053| 0| 2.5861| 53.379| [ INFO : 2022-12-03 11:56:13,530 ] - | 3| 500|0.00050583| 0| 2.5737| 53.612| [ INFO : 2022-12-03 11:57:04,873 ] - | 3| 600|0.00045625| 0| 2.5714| 53.637| [ INFO : 2022-12-03 11:57:56,229 ] - | 3| 700|0.00041134| 0| 2.5654| 53.738| [ INFO : 2022-12-03 11:58:47,349 ] - | 3| 800|0.00037068| 0| 2.5629| 53.797| [ INFO : 2022-12-03 11:59:38,587 ] - | 3| 900|0.00033391| 0| 2.5572| 53.886| [ INFO : 2022-12-03 12:00:29,940 ] - | 3| 1000|0.00030065| 0| 2.5514| 53.998| [ INFO : 2022-12-03 12:01:21,065 ] - | 3| 1100|0.00027061| 0| 2.5463| 54.11| [ INFO : 2022-12-03 12:02:12,464 ] - | 3| 1200|0.00024347| 0| 2.5448| 54.153| [ INFO : 2022-12-03 12:03:03,765 ] - | 3| 1300|0.00021898| 0| 2.5395| 54.278| [ INFO : 2022-12-03 12:03:54,967 ] - | 3| 1400|0.00019688| 0| 2.5345| 54.359| [ INFO : 2022-12-03 12:04:46,304 ] - | 3| 1500|0.00017695| 0| 2.53| 54.427| [ INFO : 2022-12-03 12:05:37,498 ] - | 3| 1600|0.00015899| 0| 2.5282| 54.465| [ INFO : 2022-12-03 12:06:28,752 ] - | 3| 1700|0.00014281| 0| 2.5244| 54.525| [ INFO : 2022-12-03 12:07:20,012 ] - | 3| 1800|0.00012823| 0| 2.5219| 54.571| [ INFO : 2022-12-03 12:08:11,061 ] - | 3| 1900|0.00011511| 0| 2.521| 54.602| [ INFO : 2022-12-03 12:09:06,858 ] - | 3| 2000|0.00010331| 0| 2.5183| 54.65| [ INFO : 2022-12-03 12:09:16,827 ] - | 3| 2016|0.00010153| 0| 2.5169| 54.668| [ INFO : 2022-12-03 12:09:16,946 ] - +----------+----------+----------+----------+----------+----------+ Namespace(dst_model='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/avg_model.pt', src_path='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models', num=2, min_epoch=0, max_epoch=65536) ['exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/model_2.pt', 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/model_3.pt'] Processing exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/model_2.pt Processing exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/model_3.pt Saving to exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/avg_model.pt Extract embeddings ... extract_embedding from /home/yoos/Documents/data/cnceleb_train/raw.list, wavs_num: 519590 extract_embedding from /home/yoos/Documents/data/eval/raw.list, wavs_num: 18772 Success eval Success cnceleb_train Embedding dir is (exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/embeddings). mean vector of enroll 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [00:00<00:00, 85128.26it/s] Score ... apply cosine scoring ... CNC-Eval-Concat.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Concat.lst: 100%|█████████████████████████████████████████████████████████████████████████████████| 3484292/3484292 [03:12<00:00, 18085.73it/s] CNC-Eval-Avg.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Avg.lst: 100%|████████████████████████████████████████████████████████████████████████████████████| 3484292/3484292 [03:00<00:00, 19316.05it/s] compute metrics (EER/minDCF) ... ---- CNC-Eval-Concat.lst.score ----- EER = 18.243 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.764 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/scores/CNC-Eval-Concat.lst.score.det.png ---- CNC-Eval-Avg.lst.score ----- EER = 16.251 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.782 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/scores/CNC-Eval-Avg.lst.score.det.png Score norm ... compute mean xvector 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2793/2793 [00:00<00:00, 8636.87it/s] compute norm score 2022-12-03 12:23:36,585 INFO get embedding ... 2022-12-03 12:23:41,758 INFO computing normed score ... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3484292/3484292 [00:03<00:00, 984563.74it/s] 2022-12-03 12:23:47,397 INFO Over! 2022-12-03 12:23:47,630 INFO get embedding ... 2022-12-03 12:23:53,196 INFO computing normed score ... 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3484292/3484292 [00:03<00:00, 1012433.99it/s] 2022-12-03 12:23:58,724 INFO Over! compute metrics ---- cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score ----- EER = 17.634 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.716 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/scores/cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score.det.png ---- cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score ----- EER = 15.827 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.695 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/scores/cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score.det.png Export the best model ... ResNet( (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (layer1): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (1): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer2): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer3): Sequential( (0): BasicBlock( (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (4): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (5): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer4): Sequential( (0): BasicBlock( (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (pool): TSTP() (seg_1): Linear(in_features=5120, out_features=256, bias=True) (seg_bn_1): Identity() (seg_2): Identity() ) Export model successfully, see exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3/models/final.zip Large margin fine-tuning ... Start training ... exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models already exists !!! [ INFO : 2022-12-03 12:24:28,177 ] - exp_dir is: exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM [ INFO : 2022-12-03 12:24:28,177 ] - <== Passed Arguments ==> [ INFO : 2022-12-03 12:24:28,178 ] - {'checkpoint': 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/model_0.pt', [ INFO : 2022-12-03 12:24:28,178 ] - 'data_type': 'raw', [ INFO : 2022-12-03 12:24:28,178 ] - 'dataloader_args': {'batch_size': 64, [ INFO : 2022-12-03 12:24:28,178 ] - 'drop_last': True, [ INFO : 2022-12-03 12:24:28,178 ] - 'num_workers': 16, [ INFO : 2022-12-03 12:24:28,178 ] - 'pin_memory': False, [ INFO : 2022-12-03 12:24:28,178 ] - 'prefetch_factor': 8}, [ INFO : 2022-12-03 12:24:28,178 ] - 'dataset_args': {'aug_prob': 0.6, [ INFO : 2022-12-03 12:24:28,178 ] - 'fbank_args': {'dither': 1.0, [ INFO : 2022-12-03 12:24:28,178 ] - 'frame_length': 25, [ INFO : 2022-12-03 12:24:28,178 ] - 'frame_shift': 10, [ INFO : 2022-12-03 12:24:28,178 ] - 'num_mel_bins': 80}, [ INFO : 2022-12-03 12:24:28,178 ] - 'num_frms': 600, [ INFO : 2022-12-03 12:24:28,178 ] - 'resample_rate': 16000, [ INFO : 2022-12-03 12:24:28,178 ] - 'shuffle': True, [ INFO : 2022-12-03 12:24:28,178 ] - 'shuffle_args': {'shuffle_size': 2500}, [ INFO : 2022-12-03 12:24:28,178 ] - 'spec_aug': False, [ INFO : 2022-12-03 12:24:28,178 ] - 'spec_aug_args': {'max_f': 8, [ INFO : 2022-12-03 12:24:28,178 ] - 'max_t': 10, [ INFO : 2022-12-03 12:24:28,178 ] - 'num_f_mask': 1, [ INFO : 2022-12-03 12:24:28,178 ] - 'num_t_mask': 1, [ INFO : 2022-12-03 12:24:28,178 ] - 'prob': 0.6}, [ INFO : 2022-12-03 12:24:28,178 ] - 'speed_perturb': True}, [ INFO : 2022-12-03 12:24:28,178 ] - 'do_lm': True, [ INFO : 2022-12-03 12:24:28,178 ] - 'enable_amp': False, [ INFO : 2022-12-03 12:24:28,178 ] - 'exp_dir': 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM', [ INFO : 2022-12-03 12:24:28,178 ] - 'gpus': [0], [ INFO : 2022-12-03 12:24:28,178 ] - 'log_batch_interval': 100, [ INFO : 2022-12-03 12:24:28,178 ] - 'loss': 'CrossEntropyLoss', [ INFO : 2022-12-03 12:24:28,178 ] - 'loss_args': {}, [ INFO : 2022-12-03 12:24:28,178 ] - 'margin_scheduler': 'MarginScheduler', [ INFO : 2022-12-03 12:24:28,178 ] - 'margin_update': {'final_margin': 0.5, [ INFO : 2022-12-03 12:24:28,178 ] - 'fix_start_epoch': 1, [ INFO : 2022-12-03 12:24:28,178 ] - 'increase_start_epoch': 1, [ INFO : 2022-12-03 12:24:28,178 ] - 'increase_type': 'exp', [ INFO : 2022-12-03 12:24:28,178 ] - 'initial_margin': 0.5, [ INFO : 2022-12-03 12:24:28,178 ] - 'update_margin': True}, [ INFO : 2022-12-03 12:24:28,178 ] - 'model': 'ResNet34', [ INFO : 2022-12-03 12:24:28,178 ] - 'model_args': {'embed_dim': 256, [ INFO : 2022-12-03 12:24:28,178 ] - 'feat_dim': 80, [ INFO : 2022-12-03 12:24:28,178 ] - 'pooling_func': 'TSTP', [ INFO : 2022-12-03 12:24:28,178 ] - 'two_emb_layer': False}, [ INFO : 2022-12-03 12:24:28,178 ] - 'model_init': None, [ INFO : 2022-12-03 12:24:28,178 ] - 'noise_data': '/home/yoos/Documents/data/musan/lmdb', [ INFO : 2022-12-03 12:24:28,178 ] - 'num_avg': 1, [ INFO : 2022-12-03 12:24:28,178 ] - 'num_epochs': 2, [ INFO : 2022-12-03 12:24:28,178 ] - 'optimizer': 'SGD', [ INFO : 2022-12-03 12:24:28,178 ] - 'optimizer_args': {'momentum': 0.9, 'nesterov': True, 'weight_decay': 0.0001}, [ INFO : 2022-12-03 12:24:28,178 ] - 'projection_args': {'easy_margin': False, [ INFO : 2022-12-03 12:24:28,178 ] - 'project_type': 'arc_margin', [ INFO : 2022-12-03 12:24:28,178 ] - 'scale': 32.0}, [ INFO : 2022-12-03 12:24:28,178 ] - 'reverb_data': '/home/yoos/Documents/data/rirs/lmdb', [ INFO : 2022-12-03 12:24:28,178 ] - 'save_epoch_interval': 1, [ INFO : 2022-12-03 12:24:28,178 ] - 'scheduler': 'ExponentialDecrease', [ INFO : 2022-12-03 12:24:28,178 ] - 'scheduler_args': {'final_lr': 2.5e-05, [ INFO : 2022-12-03 12:24:28,178 ] - 'initial_lr': 0.0001, [ INFO : 2022-12-03 12:24:28,178 ] - 'warm_from_zero': True, [ INFO : 2022-12-03 12:24:28,178 ] - 'warm_up_epoch': 1}, [ INFO : 2022-12-03 12:24:28,178 ] - 'seed': 42, [ INFO : 2022-12-03 12:24:28,178 ] - 'train_data': '/home/yoos/Documents/data/cnceleb_train/raw.list', [ INFO : 2022-12-03 12:24:28,178 ] - 'train_label': '/home/yoos/Documents/data/cnceleb_train/utt2spk'} [ INFO : 2022-12-03 12:24:28,698 ] - <== Data statistics ==> [ INFO : 2022-12-03 12:24:28,698 ] - train data num: 519590, spk num: 2793 [ INFO : 2022-12-03 12:24:28,785 ] - <== Dataloaders ==> [ INFO : 2022-12-03 12:24:28,785 ] - train dataloaders created [ INFO : 2022-12-03 12:24:28,785 ] - loader size: 8118 [ INFO : 2022-12-03 12:24:28,785 ] - <== Model ==> [ INFO : 2022-12-03 12:24:28,854 ] - speaker_model size: 6634336 [ INFO : 2022-12-03 12:24:28,854 ] - No speed perturb while doing large margin fine-tuning [ INFO : 2022-12-03 12:24:28,863 ] - ResNet( [ INFO : 2022-12-03 12:24:28,863 ] - (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (layer1): Sequential( [ INFO : 2022-12-03 12:24:28,863 ] - (0): BasicBlock( [ INFO : 2022-12-03 12:24:28,863 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,863 ] - ) [ INFO : 2022-12-03 12:24:28,863 ] - (1): BasicBlock( [ INFO : 2022-12-03 12:24:28,863 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,863 ] - ) [ INFO : 2022-12-03 12:24:28,863 ] - (2): BasicBlock( [ INFO : 2022-12-03 12:24:28,863 ] - (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,863 ] - ) [ INFO : 2022-12-03 12:24:28,863 ] - ) [ INFO : 2022-12-03 12:24:28,863 ] - (layer2): Sequential( [ INFO : 2022-12-03 12:24:28,863 ] - (0): BasicBlock( [ INFO : 2022-12-03 12:24:28,863 ] - (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,863 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,863 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential( [ INFO : 2022-12-03 12:24:28,864 ] - (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (1): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (2): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (3): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (layer3): Sequential( [ INFO : 2022-12-03 12:24:28,864 ] - (0): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential( [ INFO : 2022-12-03 12:24:28,864 ] - (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (1): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (2): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (3): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (4): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,864 ] - (5): BasicBlock( [ INFO : 2022-12-03 12:24:28,864 ] - (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,864 ] - (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,864 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,864 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - (layer4): Sequential( [ INFO : 2022-12-03 12:24:28,865 ] - (0): BasicBlock( [ INFO : 2022-12-03 12:24:28,865 ] - (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - (shortcut): Sequential( [ INFO : 2022-12-03 12:24:28,865 ] - (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - (1): BasicBlock( [ INFO : 2022-12-03 12:24:28,865 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - (2): BasicBlock( [ INFO : 2022-12-03 12:24:28,865 ] - (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) [ INFO : 2022-12-03 12:24:28,865 ] - (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) [ INFO : 2022-12-03 12:24:28,865 ] - (shortcut): Sequential() [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - (pool): TSTP() [ INFO : 2022-12-03 12:24:28,865 ] - (seg_1): Linear(in_features=5120, out_features=256, bias=True) [ INFO : 2022-12-03 12:24:28,865 ] - (seg_bn_1): Identity() [ INFO : 2022-12-03 12:24:28,865 ] - (seg_2): Identity() [ INFO : 2022-12-03 12:24:28,865 ] - (projection): ArcMarginProduct( [ INFO : 2022-12-03 12:24:28,865 ] - in_features=256, out_features=8379, scale=32.0, [ INFO : 2022-12-03 12:24:28,865 ] - margin=0.0, easy_margin=False [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:28,865 ] - ) [ INFO : 2022-12-03 12:24:29,063 ] - Load checkpoint: exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/model_0.pt [ INFO : 2022-12-03 12:24:29,063 ] - start_epoch: 1 [ INFO : 2022-12-03 12:24:29,080 ] - <== Loss ==> [ INFO : 2022-12-03 12:24:29,080 ] - loss criterion is: CrossEntropyLoss [ INFO : 2022-12-03 12:24:29,080 ] - <== Optimizer ==> [ INFO : 2022-12-03 12:24:29,080 ] - optimizer is: SGD [ INFO : 2022-12-03 12:24:29,080 ] - <== Scheduler ==> [ INFO : 2022-12-03 12:24:29,080 ] - scheduler is: ExponentialDecrease [ INFO : 2022-12-03 12:24:29,080 ] - <== MarginScheduler ==> [ INFO : 2022-12-03 12:24:29,082 ] - <========== Training process ==========> [ INFO : 2022-12-03 12:24:29,082 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 12:24:29,082 ] - | Epoch| Batch| Lr| Margin| Loss| Acc| [ INFO : 2022-12-03 12:24:29,083 ] - +----------+----------+----------+----------+----------+----------+ [ INFO : 2022-12-03 12:24:39,091 ] - Reducer buckets have been rebuilt in this iteration. [ INFO : 2022-12-03 12:25:17,476 ] - | 1| 100|1.2092e-06| 0.5| 15.795| 0| [ INFO : 2022-12-03 12:25:56,673 ] - | 1| 200| 2.41e-06| 0.5| 15.792| 0| [ INFO : 2022-12-03 12:26:35,697 ] - | 1| 300|3.5903e-06| 0.5| 15.777| 0| [ INFO : 2022-12-03 12:27:12,757 ] - | 1| 400|4.7504e-06| 0.5| 15.763| 0| [ INFO : 2022-12-03 12:27:49,827 ] - | 1| 500|5.8904e-06| 0.5| 15.76| 0| [ INFO : 2022-12-03 12:28:27,082 ] - | 1| 600|7.0108e-06| 0.5| 15.752| 0| [ INFO : 2022-12-03 12:29:04,196 ] - | 1| 700|8.1116e-06| 0.5| 15.748| 0| [ INFO : 2022-12-03 12:29:41,307 ] - | 1| 800|9.1933e-06| 0.5| 15.749| 0| [ INFO : 2022-12-03 12:30:18,427 ] - | 1| 900|1.0256e-05| 0.5| 15.741| 0| [ INFO : 2022-12-03 12:30:55,527 ] - | 1| 1000| 1.13e-05| 0.5| 15.735| 0| [ INFO : 2022-12-03 12:31:32,691 ] - | 1| 1100|1.2325e-05| 0.5| 15.727| 0| [ INFO : 2022-12-03 12:32:10,221 ] - | 1| 1200|1.3332e-05| 0.5| 15.721| 0| [ INFO : 2022-12-03 12:32:48,126 ] - | 1| 1300|1.4322e-05| 0.5| 15.719| 0| [ INFO : 2022-12-03 12:33:25,951 ] - | 1| 1400|1.5293e-05| 0.5| 15.712| 0| [ INFO : 2022-12-03 12:34:03,441 ] - | 1| 1500|1.6247e-05| 0.5| 15.71| 0| [ INFO : 2022-12-03 12:34:41,124 ] - | 1| 1600|1.7183e-05| 0.5| 15.706| 0| [ INFO : 2022-12-03 12:35:18,284 ] - | 1| 1700|1.8103e-05| 0.5| 15.702| 0| [ INFO : 2022-12-03 12:35:55,524 ] - | 1| 1800|1.9005e-05| 0.5| 15.697| 0| [ INFO : 2022-12-03 12:36:32,623 ] - | 1| 1900|1.9891e-05| 0.5| 15.692| 0| [ INFO : 2022-12-03 12:37:09,714 ] - | 1| 2000| 2.076e-05| 0.5| 15.688| 0| [ INFO : 2022-12-03 12:37:46,842 ] - | 1| 2100|2.1614e-05| 0.5| 15.687| 0| [ INFO : 2022-12-03 12:38:24,031 ] - | 1| 2200|2.2451e-05| 0.5| 15.683| 0| [ INFO : 2022-12-03 12:39:01,326 ] - | 1| 2300|2.3272e-05| 0.5| 15.683| 0| [ INFO : 2022-12-03 12:39:38,683 ] - | 1| 2400|2.4078e-05| 0.5| 15.681| 0| [ INFO : 2022-12-03 12:40:15,795 ] - | 1| 2500|2.4869e-05| 0.5| 15.677| 0| [ INFO : 2022-12-03 12:40:52,879 ] - | 1| 2600|2.5644e-05| 0.5| 15.67| 0| [ INFO : 2022-12-03 12:41:29,986 ] - | 1| 2700|2.6404e-05| 0.5| 15.669| 0| [ INFO : 2022-12-03 12:42:07,136 ] - | 1| 2800| 2.715e-05| 0.5| 15.668| 0| [ INFO : 2022-12-03 12:42:44,340 ] - | 1| 2900| 2.788e-05| 0.5| 15.664| 0| [ INFO : 2022-12-03 12:43:21,564 ] - | 1| 3000|2.8597e-05| 0.5| 15.658| 0| [ INFO : 2022-12-03 12:43:58,836 ] - | 1| 3100|2.9299e-05| 0.5| 15.652| 0| [ INFO : 2022-12-03 12:44:35,963 ] - | 1| 3200|2.9988e-05| 0.5| 15.648| 0| [ INFO : 2022-12-03 12:45:13,068 ] - | 1| 3300|3.0662e-05| 0.5| 15.645| 0| [ INFO : 2022-12-03 12:45:50,171 ] - | 1| 3400|3.1323e-05| 0.5| 15.641| 0| [ INFO : 2022-12-03 12:46:27,303 ] - | 1| 3500| 3.197e-05| 0.5| 15.641| 0| [ INFO : 2022-12-03 12:47:04,441 ] - | 1| 3600|3.2604e-05| 0.5| 15.635| 0| [ INFO : 2022-12-03 12:47:41,674 ] - | 1| 3700|3.3225e-05| 0.5| 15.633| 0| [ INFO : 2022-12-03 12:48:18,785 ] - | 1| 3800|3.3833e-05| 0.5| 15.629| 0| [ INFO : 2022-12-03 12:48:55,898 ] - | 1| 3900|3.4429e-05| 0.5| 15.626| 0| [ INFO : 2022-12-03 12:49:33,019 ] - | 1| 4000|3.5012e-05| 0.5| 15.625| 0| [ INFO : 2022-12-03 12:50:10,184 ] - | 1| 4100|3.5582e-05| 0.5| 15.621| 0| [ INFO : 2022-12-03 12:50:47,449 ] - | 1| 4200| 3.614e-05| 0.5| 15.615| 0| [ INFO : 2022-12-03 12:51:24,748 ] - | 1| 4300|3.6686e-05| 0.5| 15.61| 0| [ INFO : 2022-12-03 12:52:01,859 ] - | 1| 4400| 3.722e-05| 0.5| 15.605| 0| [ INFO : 2022-12-03 12:52:38,978 ] - | 1| 4500|3.7743e-05| 0.5| 15.599| 0| [ INFO : 2022-12-03 12:53:16,094 ] - | 1| 4600|3.8254e-05| 0.5| 15.597| 0| [ INFO : 2022-12-03 12:53:53,277 ] - | 1| 4700|3.8753e-05| 0.5| 15.593| 0| [ INFO : 2022-12-03 12:54:30,459 ] - | 1| 4800|3.9242e-05| 0.5| 15.589| 0| [ INFO : 2022-12-03 12:55:07,761 ] - | 1| 4900|3.9719e-05| 0.5| 15.585| 0| [ INFO : 2022-12-03 12:55:44,874 ] - | 1| 5000|4.0185e-05| 0.5| 15.581| 0| [ INFO : 2022-12-03 12:56:21,977 ] - | 1| 5100| 4.064e-05| 0.5| 15.575| 0| [ INFO : 2022-12-03 12:56:59,107 ] - | 1| 5200|4.1085e-05| 0.5| 15.57| 0| [ INFO : 2022-12-03 12:57:36,287 ] - | 1| 5300|4.1519e-05| 0.5| 15.565| 0| [ INFO : 2022-12-03 12:58:13,498 ] - | 1| 5400|4.1943e-05| 0.5| 15.562| 0| [ INFO : 2022-12-03 12:58:50,741 ] - | 1| 5500|4.2357e-05| 0.5| 15.558| 0| [ INFO : 2022-12-03 12:59:27,974 ] - | 1| 5600| 4.276e-05| 0.5| 15.552| 0| [ INFO : 2022-12-03 13:00:05,130 ] - | 1| 5700|4.3154e-05| 0.5| 15.548| 0| [ INFO : 2022-12-03 13:00:42,269 ] - | 1| 5800|4.3538e-05| 0.5| 15.544| 0| [ INFO : 2022-12-03 13:01:19,415 ] - | 1| 5900|4.3912e-05| 0.5| 15.54| 0| [ INFO : 2022-12-03 13:01:56,603 ] - | 1| 6000|4.4277e-05| 0.5| 15.538| 0| [ INFO : 2022-12-03 13:02:33,857 ] - | 1| 6100|4.4632e-05| 0.5| 15.535| 0| [ INFO : 2022-12-03 13:03:11,151 ] - | 1| 6200|4.4978e-05| 0.5| 15.531| 0| [ INFO : 2022-12-03 13:03:48,280 ] - | 1| 6300|4.5315e-05| 0.5| 15.526|0.00024802| [ INFO : 2022-12-03 13:04:25,424 ] - | 1| 6400|4.5643e-05| 0.5| 15.523|0.00024414| [ INFO : 2022-12-03 13:05:02,539 ] - | 1| 6500|4.5962e-05| 0.5| 15.517|0.00024038| [ INFO : 2022-12-03 13:05:39,697 ] - | 1| 6600|4.6273e-05| 0.5| 15.514|0.00023674| [ INFO : 2022-12-03 13:06:16,864 ] - | 1| 6700|4.6575e-05| 0.5| 15.51|0.00023321| [ INFO : 2022-12-03 13:06:54,165 ] - | 1| 6800|4.6868e-05| 0.5| 15.505|0.00022978| [ INFO : 2022-12-03 13:07:31,301 ] - | 1| 6900|4.7153e-05| 0.5| 15.501|0.00022645| [ INFO : 2022-12-03 13:08:08,424 ] - | 1| 7000| 4.743e-05| 0.5| 15.496|0.00044643| [ INFO : 2022-12-03 13:08:45,649 ] - | 1| 7100|4.7698e-05| 0.5| 15.491|0.00044014| [ INFO : 2022-12-03 13:09:22,824 ] - | 1| 7200|4.7959e-05| 0.5| 15.487|0.00043403| [ INFO : 2022-12-03 13:09:59,975 ] - | 1| 7300|4.8212e-05| 0.5| 15.484|0.00042808| [ INFO : 2022-12-03 13:10:37,322 ] - | 1| 7400|4.8457e-05| 0.5| 15.48| 0.0004223| [ INFO : 2022-12-03 13:11:14,453 ] - | 1| 7500|4.8694e-05| 0.5| 15.477| 0.000625| [ INFO : 2022-12-03 13:11:51,567 ] - | 1| 7600|4.8924e-05| 0.5| 15.474|0.00061678| [ INFO : 2022-12-03 13:12:28,713 ] - | 1| 7700|4.9146e-05| 0.5| 15.469|0.00081169| [ INFO : 2022-12-03 13:13:05,893 ] - | 1| 7800|4.9362e-05| 0.5| 15.467|0.00080128| [ INFO : 2022-12-03 13:13:43,008 ] - | 1| 7900|4.9569e-05| 0.5| 15.462|0.00098892| [ INFO : 2022-12-03 13:14:20,157 ] - | 1| 8000| 4.977e-05| 0.5| 15.459| 0.0011719| [ INFO : 2022-12-03 13:14:58,481 ] - | 1| 8100|4.9964e-05| 0.5| 15.455| 0.0013503| [ INFO : 2022-12-03 13:15:03,201 ] - | 1| 8112|4.9987e-05| 0.5| 15.454| 0.0013483| [ INFO : 2022-12-03 13:15:47,516 ] - | 2| 100|4.9579e-05| 0.5| 15.1| 0.046875| [ INFO : 2022-12-03 13:16:24,639 ] - | 2| 200|4.9158e-05| 0.5| 15.123| 0.03125| [ INFO : 2022-12-03 13:17:01,750 ] - | 2| 300| 4.874e-05| 0.5| 15.141| 0.020833| [ INFO : 2022-12-03 13:17:38,892 ] - | 2| 400|4.8325e-05| 0.5| 15.114| 0.027344| [ INFO : 2022-12-03 13:18:16,064 ] - | 2| 500|4.7914e-05| 0.5| 15.115| 0.025| [ INFO : 2022-12-03 13:18:53,341 ] - | 2| 600|4.7507e-05| 0.5| 15.117| 0.023438| [ INFO : 2022-12-03 13:19:30,500 ] - | 2| 700|4.7103e-05| 0.5| 15.124| 0.022321| [ INFO : 2022-12-03 13:20:07,633 ] - | 2| 800|4.6703e-05| 0.5| 15.117| 0.021484| [ INFO : 2022-12-03 13:20:44,741 ] - | 2| 900|4.6306e-05| 0.5| 15.116| 0.020833| [ INFO : 2022-12-03 13:21:22,610 ] - | 2| 1000|4.5912e-05| 0.5| 15.122| 0.021875| [ INFO : 2022-12-03 13:22:01,061 ] - | 2| 1100|4.5522e-05| 0.5| 15.124| 0.024148| [ INFO : 2022-12-03 13:22:38,436 ] - | 2| 1200|4.5135e-05| 0.5| 15.116| 0.027344| [ INFO : 2022-12-03 13:23:15,551 ] - | 2| 1300|4.4751e-05| 0.5| 15.122| 0.028846| [ INFO : 2022-12-03 13:23:52,642 ] - | 2| 1400| 4.437e-05| 0.5| 15.111| 0.032366| [ INFO : 2022-12-03 13:24:29,746 ] - | 2| 1500|4.3993e-05| 0.5| 15.105| 0.032292| [ INFO : 2022-12-03 13:25:06,861 ] - | 2| 1600|4.3619e-05| 0.5| 15.099| 0.033203| [ INFO : 2022-12-03 13:25:43,951 ] - | 2| 1700|4.3248e-05| 0.5| 15.096| 0.034926| [ INFO : 2022-12-03 13:26:21,227 ] - | 2| 1800|4.2881e-05| 0.5| 15.092| 0.034722| [ INFO : 2022-12-03 13:26:58,335 ] - | 2| 1900|4.2516e-05| 0.5| 15.087| 0.033717| [ INFO : 2022-12-03 13:27:35,432 ] - | 2| 2000|4.2154e-05| 0.5| 15.081| 0.034375| [ INFO : 2022-12-03 13:28:12,524 ] - | 2| 2100|4.1796e-05| 0.5| 15.077| 0.03497| [ INFO : 2022-12-03 13:28:49,603 ] - | 2| 2200|4.1441e-05| 0.5| 15.073| 0.034801| [ INFO : 2022-12-03 13:29:26,707 ] - | 2| 2300|4.1088e-05| 0.5| 15.07| 0.035326| [ INFO : 2022-12-03 13:30:03,928 ] - | 2| 2400|4.0739e-05| 0.5| 15.067| 0.035156| [ INFO : 2022-12-03 13:30:41,018 ] - | 2| 2500|4.0393e-05| 0.5| 15.063| 0.036875| [ INFO : 2022-12-03 13:31:18,100 ] - | 2| 2600|4.0049e-05| 0.5| 15.062| 0.038462| [ INFO : 2022-12-03 13:31:55,211 ] - | 2| 2700|3.9709e-05| 0.5| 15.058| 0.038773| [ INFO : 2022-12-03 13:32:32,290 ] - | 2| 2800|3.9371e-05| 0.5| 15.059| 0.041853| [ INFO : 2022-12-03 13:33:09,417 ] - | 2| 2900|3.9036e-05| 0.5| 15.054| 0.043642| [ INFO : 2022-12-03 13:33:46,530 ] - | 2| 3000|3.8705e-05| 0.5| 15.049| 0.04375| [ INFO : 2022-12-03 13:34:23,751 ] - | 2| 3100|3.8375e-05| 0.5| 15.046| 0.043851| [ INFO : 2022-12-03 13:35:00,870 ] - | 2| 3200|3.8049e-05| 0.5| 15.044| 0.04541| [ INFO : 2022-12-03 13:35:37,967 ] - | 2| 3300|3.7726e-05| 0.5| 15.041| 0.044981| [ INFO : 2022-12-03 13:36:15,067 ] - | 2| 3400|3.7405e-05| 0.5| 15.038| 0.046415| [ INFO : 2022-12-03 13:36:52,169 ] - | 2| 3500|3.7087e-05| 0.5| 15.031| 0.049554| [ INFO : 2022-12-03 13:37:29,277 ] - | 2| 3600|3.6772e-05| 0.5| 15.028| 0.049913| [ INFO : 2022-12-03 13:38:06,563 ] - | 2| 3700|3.6459e-05| 0.5| 15.026| 0.049831| [ INFO : 2022-12-03 13:38:43,687 ] - | 2| 3800|3.6149e-05| 0.5| 15.023| 0.05222| [ INFO : 2022-12-03 13:39:20,798 ] - | 2| 3900|3.5842e-05| 0.5| 15.019| 0.053686| [ INFO : 2022-12-03 13:39:57,922 ] - | 2| 4000|3.5537e-05| 0.5| 15.018| 0.054297| [ INFO : 2022-12-03 13:40:35,063 ] - | 2| 4100|3.5235e-05| 0.5| 15.014| 0.054878| [ INFO : 2022-12-03 13:41:12,167 ] - | 2| 4200|3.4935e-05| 0.5| 15.011| 0.055804| [ INFO : 2022-12-03 13:41:49,426 ] - | 2| 4300|3.4638e-05| 0.5| 15.009| 0.05814| [ INFO : 2022-12-03 13:42:26,537 ] - | 2| 4400|3.4344e-05| 0.5| 15.005| 0.058594| [ INFO : 2022-12-03 13:43:03,676 ] - | 2| 4500|3.4052e-05| 0.5| 15.002| 0.058333| [ INFO : 2022-12-03 13:43:40,948 ] - | 2| 4600|3.3762e-05| 0.5| 15| 0.059783| [ INFO : 2022-12-03 13:44:18,093 ] - | 2| 4700|3.3475e-05| 0.5| 14.996| 0.06117| [ INFO : 2022-12-03 13:44:55,232 ] - | 2| 4800|3.3191e-05| 0.5| 14.993| 0.062174| [ INFO : 2022-12-03 13:45:32,471 ] - | 2| 4900|3.2908e-05| 0.5| 14.99| 0.063457| [ INFO : 2022-12-03 13:46:09,567 ] - | 2| 5000|3.2629e-05| 0.5| 14.986| 0.064375| [ INFO : 2022-12-03 13:46:46,667 ] - | 2| 5100|3.2351e-05| 0.5| 14.983| 0.064645| [ INFO : 2022-12-03 13:47:23,770 ] - | 2| 5200|3.2076e-05| 0.5| 14.981| 0.065204| [ INFO : 2022-12-03 13:48:00,867 ] - | 2| 5300|3.1803e-05| 0.5| 14.978| 0.066038| [ INFO : 2022-12-03 13:48:37,963 ] - | 2| 5400|3.1533e-05| 0.5| 14.976| 0.067998| [ INFO : 2022-12-03 13:49:15,105 ] - | 2| 5500|3.1265e-05| 0.5| 14.974| 0.069318| [ INFO : 2022-12-03 13:49:52,281 ] - | 2| 5600|3.0999e-05| 0.5| 14.969| 0.068638| [ INFO : 2022-12-03 13:50:29,371 ] - | 2| 5700|3.0736e-05| 0.5| 14.965| 0.069079| [ INFO : 2022-12-03 13:51:06,445 ] - | 2| 5800|3.0474e-05| 0.5| 14.963| 0.070582| [ INFO : 2022-12-03 13:51:43,515 ] - | 2| 5900|3.0215e-05| 0.5| 14.96| 0.071239| [ INFO : 2022-12-03 13:52:20,829 ] - | 2| 6000|2.9958e-05| 0.5| 14.958| 0.072656| [ INFO : 2022-12-03 13:52:58,265 ] - | 2| 6100|2.9704e-05| 0.5| 14.956| 0.073514| [ INFO : 2022-12-03 13:53:35,874 ] - | 2| 6200|2.9451e-05| 0.5| 14.953| 0.075353| [ INFO : 2022-12-03 13:54:13,296 ] - | 2| 6300|2.9201e-05| 0.5| 14.951| 0.075645| [ INFO : 2022-12-03 13:54:50,684 ] - | 2| 6400|2.8952e-05| 0.5| 14.949| 0.077393| [ INFO : 2022-12-03 13:55:28,138 ] - | 2| 6500|2.8706e-05| 0.5| 14.945| 0.078365| [ INFO : 2022-12-03 13:56:05,482 ] - | 2| 6600|2.8462e-05| 0.5| 14.941| 0.079309| [ INFO : 2022-12-03 13:56:42,636 ] - | 2| 6700| 2.822e-05| 0.5| 14.941| 0.080224| [ INFO : 2022-12-03 13:57:19,848 ] - | 2| 6800| 2.798e-05| 0.5| 14.939| 0.081342| [ INFO : 2022-12-03 13:57:56,931 ] - | 2| 6900|2.7742e-05| 0.5| 14.938| 0.082428| [ INFO : 2022-12-03 13:58:34,017 ] - | 2| 7000|2.7506e-05| 0.5| 14.934| 0.083036| [ INFO : 2022-12-03 13:59:11,116 ] - | 2| 7100|2.7273e-05| 0.5| 14.931| 0.083187| [ INFO : 2022-12-03 13:59:48,193 ] - | 2| 7200|2.7041e-05| 0.5| 14.927| 0.083984| [ INFO : 2022-12-03 14:00:25,281 ] - | 2| 7300|2.6811e-05| 0.5| 14.924| 0.084546| [ INFO : 2022-12-03 14:01:02,510 ] - | 2| 7400|2.6583e-05| 0.5| 14.92| 0.085726| [ INFO : 2022-12-03 14:01:39,613 ] - | 2| 7500|2.6357e-05| 0.5| 14.916| 0.087083| [ INFO : 2022-12-03 14:02:16,702 ] - | 2| 7600|2.6133e-05| 0.5| 14.914| 0.087993| [ INFO : 2022-12-03 14:02:53,781 ] - | 2| 7700|2.5911e-05| 0.5| 14.914| 0.088271| [ INFO : 2022-12-03 14:03:30,892 ] - | 2| 7800| 2.569e-05| 0.5| 14.912| 0.087941| [ INFO : 2022-12-03 14:04:07,985 ] - | 2| 7900|2.5472e-05| 0.5| 14.91| 0.089399| [ INFO : 2022-12-03 14:04:45,065 ] - | 2| 8000|2.5255e-05| 0.5| 14.908| 0.089648| [ INFO : 2022-12-03 14:05:22,119 ] - | 2| 8100|2.5041e-05| 0.5| 14.905| 0.091242| [ INFO : 2022-12-03 14:05:26,870 ] - | 2| 8112|2.5015e-05| 0.5| 14.905| 0.091493| [ INFO : 2022-12-03 14:05:26,910 ] - +----------+----------+----------+----------+----------+----------+ Namespace(dst_model='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/avg_model.pt', src_path='exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models', num=1, min_epoch=0, max_epoch=65536) ['exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/model_2.pt'] Processing exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/model_2.pt Saving to exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/avg_model.pt Extract embeddings ... extract_embedding from /home/yoos/Documents/data/cnceleb_train/raw.list, wavs_num: 519590 extract_embedding from /home/yoos/Documents/data/eval/raw.list, wavs_num: 18772 Fail eval Success cnceleb_train Embedding dir is (exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/embeddings). mean vector of enroll 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 196/196 [00:00<00:00, 85669.40it/s] Score ... apply cosine scoring ... CNC-Eval-Concat.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Concat.lst: 0%|▏ | 10420/3484292 [00:00<03:02, 19050.51it/s] Traceback (most recent call last): File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 96, in fire.Fire(main) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, *kwargs) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 92, in main trials_cosine_score(eval_scp_path, store_score_dir, mean_vec_path, trials) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 62, in trials_cosine_score emb1, emb2 = emb_dict[segs[0]], emb_dict[segs[1]] KeyError: 'test/id00922-interview-05-003.wav' CNC-Eval-Avg.lst Calculate mean statistics from exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/embeddings/cnceleb_train/xvector.scp. scoring trial CNC-Eval-Avg.lst: 0%|▎ | 10420/3484292 [00:00<03:02, 19064.72it/s] Traceback (most recent call last): File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 96, in fire.Fire(main) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(varargs, **kwargs) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 92, in main trials_cosine_score(eval_scp_path, store_score_dir, mean_vec_path, trials) File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/score.py", line 62, in trials_cosine_score emb1, emb2 = emb_dict[segs[0]], emb_dict[segs[1]] KeyError: 'test/id00922-interview-05-003.wav' compute metrics (EER/minDCF) ... ---- CNC-Eval-Concat.lst.score ----- EER = 16.675 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.467 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/scores/CNC-Eval-Concat.lst.score.det.png ---- CNC-Eval-Avg.lst.score ----- EER = 13.333 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.552 compute DET curve ... DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/scores/CNC-Eval-Avg.lst.score.det.png Score norm ... compute mean xvector 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2793/2793 [00:00<00:00, 9603.17it/s] compute norm score 2022-12-03 14:24:30,184 INFO get embedding ... 2022-12-03 14:24:30,335 INFO computing normed score ... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10420/10420 [00:00<00:00, 933441.14it/s] 2022-12-03 14:24:31,491 INFO Over! 2022-12-03 14:24:31,665 INFO get embedding ... 2022-12-03 14:24:31,812 INFO computing normed score ... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10420/10420 [00:00<00:00, 930777.29it/s] 2022-12-03 14:24:33,029 INFO Over! compute metrics ---- cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score ----- EER = 18.856 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.486 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/scores/cnceleb_train_asnorm300_CNC-Eval-Concat.lst.score.det.png ---- cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score ----- EER = 13.936 minDCF (p_target:0.01 c_miss:1 c_fa:1) = 0.438 DET curve saved in exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/scores/cnceleb_train_asnorm300_CNC-Eval-Avg.lst.score.det.png Export the best model ... ResNet( (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (layer1): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (1): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer2): Sequential( (0): BasicBlock( (conv1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer3): Sequential( (0): BasicBlock( (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (3): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (4): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (5): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (layer4): Sequential( (0): BasicBlock( (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) (2): BasicBlock( (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (shortcut): Sequential() ) ) (pool): TSTP() (seg_1): Linear(in_features=5120, out_features=256, bias=True) (seg_bn_1): Identity() (seg_2): Identity() ) Export model successfully, see exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/models/final.zip

cdliang11 commented 1 year ago

sorry, Why is it successful to extract eval embedding in the first stage of training (without LM), but it fails to extract eval embedding in the LM stage? Is there any difference between them?

Hi, the log shows that the program did not successfully extract the embedding of the eval dataset. You should check 'exp/ResNet34-TSTP-emb256-fbank80-num_frms200-aug0.6-spTrue-saFalse-ArcMargin-SGD-epoch3-LM/embeddings/eval/log/**.log'. I guess it's an OOM error, becuase the num_frms is 600 in the stage of LM.

liyunlongaaa commented 1 year ago

d00922-interview-05-003.

WOW, thank you!!! you are so awesome！

Traceback (most recent call last):
  File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/extract.py", line 93, in <module>
    fire.Fire(extract)
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/yoos/Documents/code/wespeaker/examples/cnceleb/v2/wespeaker/bin/extract.py", line 46, in extract
    model.to(device).eval()
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 899, in to
    return self._apply(convert)
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 570, in _apply
    module._apply(fn)
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 593, in _apply
    param_applied = fn(param)
  File "/home/yoos/miniconda3/envs/wespeaker/lib/python3.9/site-packages/torch/nn/modules/module.py", line 897, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

How can I modify the configuration to avoid this situation? My GPU is 3090 24G

cdliang11 commented 1 year ago

How can I modify the configuration to avoid this situation? My GPU is 3090 24G

https://github.com/wenet-e2e/wespeaker/blob/7a7c3b11c3888d0e5bb7e64d3a50efa17ac56397/examples/cnceleb/v2/local/extract_cnc.sh#L29

Currently, the embedding of the cnceleb_train has been successfully extracted. You can try to modify the code in 'extract_cnc.sh' to extract only the embedding of eval.

liyunlongaaa commented 1 year ago

How can I modify the configuration to avoid this situation? My GPU is 3090 24G

https://github.com/wenet-e2e/wespeaker/blob/7a7c3b11c3888d0e5bb7e64d3a50efa17ac56397/examples/cnceleb/v2/local/extract_cnc.sh#L29

Currently, the embedding of the cnceleb_train has been successfully extracted. You can try to modify the code in 'extract_cnc.sh' to extract only the embedding of eval.

wow! thank you ! it work. But it is a bit a little cumbersome，if I want to they success together, How should I do?

cdliang11 commented 1 year ago

wow! thank you ! it work. But it is a bit a little cumbersome，if I want to they success together, How should I do?

https://github.com/wenet-e2e/wespeaker/blob/7a7c3b11c3888d0e5bb7e64d3a50efa17ac56397/examples/cnceleb/v2/local/extract_cnc.sh#L33

Maybe you can set batch_size_array=(8 1).

liyunlongaaa commented 1 year ago

It doesn't work...Why do these two extraction processes take up gpu memory at the same time?

liyunlongaaa commented 1 year ago

It doesn't work...Why do these two extraction processes take up gpu memory at the same time?

oh, I remove the '&' in 的 script, It work ! thank you very much !