search

microsoft / Semi-supervised-learning

A Unified Semi-Supervised Learning Codebase (NeurIPS'22)
https://usb.readthedocs.io
MIT License
1.37k stars 182 forks source link

The audio dataset urbansound8k test is abnormal #173

Closed yichenwang231 closed 1 year ago

yichenwang231 commented 1 year ago

Bug

When I tested UrbanSound8k, I set up 8 epochs and 4 epochs, respectively, and they were the best models stored in the fourth epoch according to the validation set, the losses and cf_mat of the first four epochs are exactly the same,but the results of loading the optimal model at the end of the run are different for the test set .

The logs are as follows：

8epoch； Semi-supervised-learning-main$ python train.py --c config/usb_audio/supervised/supervised_urbansound8k_100_0.yaml train.py:185: UserWarning: You have chosen to seed training. This will turn on the CUDNN deterministic setting, which can slow down your training considerably! You may see unexpected behavior when restarting from checkpoints. warnings.warn('You have chosen to seed training. ' [2023-10-22 14:25:27,612 INFO] Use GPU: None for training /media/ubuntu20/D/wyc/fuxian/Semi-supervised-learning-main/semilearn/datasets/utils.py:38: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. data, targets = np.array(data), np.array(targets) [2023-10-22 14:25:29,227 INFO] unlabeled data number: 7079, labeled data number 100 [2023-10-22 14:25:29,227 INFO] Create train and test data loaders [2023-10-22 14:26:09,317 INFO] [!] data loader keys: dict_keys(['train_lb', 'train_ulb', 'eval', 'test']) [2023-10-22 14:26:29,943 INFO] Create optimizer and scheduler [2023-10-22 14:26:29,945 INFO] Number of Trainable Params: 94969994 [2023-10-22 14:26:31,640 INFO] Arguments: Namespace(algorithm='supervised', amp=False, batch_size=8, c='config/usb_audio/supervised/supervised_urbansound8k_100_0.yaml', clip=0.0, clip_grad=0, crop_ratio=0.875, data_dir='./data', dataset='urbansound8k', dist_backend='nccl', dist_url='tcp://127.0.0.1:29980', distributed=False, ema_m=0.0, epoch=8, eval_batch_size=16, gpu=None, imb_algorithm=None, img_size=32, include_lb_to_ulb=True, layer_decay=0.75, lb_dest_len=100, lb_imb_ratio=1, load_path='./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth', lr=5e-05, max_length=512, max_length_seconds=4.0, momentum=0.9, multiprocessing_distributed=False, net='hubert_base', net_from_name=False, num_classes=10, num_eval_iter=2048, num_labels=100, num_log_iter=256, num_train_iter=8192, num_warmup_iter=5120, num_workers=4, optim='AdamW', overwrite=True, pretrain_path='', rank=0, resume=False, sample_rate=16000, save_dir='./saved_models/usb_audio', save_name='supervised_urbansound8k_100_0', seed=0, train_sampler='RandomSampler', ulb_dest_len=7079, ulb_imb_ratio=1, ulb_loss_ratio=1.0, ulb_num_labels=None, uratio=1, use_aim=False, use_cat=False, use_pretrain=False, use_tensorboard=True, use_wandb=False, weight_decay=2e-05, world_size=1) [2023-10-22 14:26:31,641 INFO] Resume load path ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth does not exist [2023-10-22 14:26:31,641 INFO] Model training [2023-10-22 14:26:48,589 INFO] 256 iteration USE_EMA: False, train/sup_loss: 2.2898, train/run_time: 0.0554, lr: 0.0000, train/prefecth_time: 0.0014 [2023-10-22 14:27:05,295 INFO] 512 iteration USE_EMA: False, train/sup_loss: 2.2459, train/run_time: 0.0535, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:27:21,938 INFO] 768 iteration USE_EMA: False, train/sup_loss: 2.0217, train/run_time: 0.0554, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:27:38,534 INFO] 1024 iteration USE_EMA: False, train/sup_loss: 1.6001, train/run_time: 0.0537, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:27:55,515 INFO] 1280 iteration USE_EMA: False, train/sup_loss: 1.1751, train/run_time: 0.0539, lr: 0.0000, train/prefecth_time: 0.0014 [2023-10-22 14:28:12,193 INFO] 1536 iteration USE_EMA: False, train/sup_loss: 0.6881, train/run_time: 0.0555, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:28:28,843 INFO] 1792 iteration USE_EMA: False, train/sup_loss: 0.4191, train/run_time: 0.0575, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:28:45,480 INFO] validating... [2023-10-22 14:28:47,310 INFO] confusion matrix: [[0.19 0. 0.11 0.07 0.06 0.16

  1. 0.35 0. 0.06 ] [0.15625 0.53125 0. 0.125 0.125 0.
  2. 0.0625 0. 0. ] [0. 0.04 0.59 0.09 0.06 0.
  3. 0.01 0.12 0.09 ] [0.02 0.1 0.1 0.41 0.02 0.04 0.13 0.08 0.01 0.09 ] [0.12 0.05 0. 0. 0.37 0.06 0.11 0.25 0.02 0.02 ] [0.26966292 0. 0.01123596 0.03370787 0. 0.24719101
  4. 0.42696629 0.01123596 0. ] [0. 0.12903226 0. 0.06451613 0. 0. 0.74193548 0.06451613 0. 0. ] [0.06097561 0. 0. 0. 0.01219512 0.06097561
  5. 0.86585366 0. 0. ] [0. 0.02439024 0. 0. 0.01219512 0.
  6. 0.03658537 0.92682927 0. ] [0. 0.06 0.18 0.01 0.02 0.
    1. 0.12 0.61 ]] [2023-10-22 14:28:48,248 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:28:49,165 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/model_best.pth [2023-10-22 14:28:49,165 INFO] 2048 iteration, USE_EMA: False, train/sup_loss: 0.3801, train/run_time: 0.0553, eval/loss: 1.9100, eval/top-1-acc: 0.5221, eval/balanced_acc: 0.5483, eval/precision: 0.5193, eval/recall: 0.5483, eval/F1: 0.5103, lr: 0.0000, train/prefecth_time: 0.0010 BEST_EVAL_ACC: 0.5221, at 2048 iters [2023-10-22 14:29:06,224 INFO] 2304 iteration USE_EMA: False, train/sup_loss: 0.0385, train/run_time: 0.0575, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:29:22,958 INFO] 2560 iteration USE_EMA: False, train/sup_loss: 1.6380, train/run_time: 0.0516, lr: 0.0000, train/prefecth_time: 0.0016 [2023-10-22 14:29:39,666 INFO] 2816 iteration USE_EMA: False, train/sup_loss: 0.0323, train/run_time: 0.0556, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:29:56,411 INFO] 3072 iteration USE_EMA: False, train/sup_loss: 0.6749, train/run_time: 0.0534, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:30:13,461 INFO] 3328 iteration USE_EMA: False, train/sup_loss: 0.0032, train/run_time: 0.0578, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:30:30,235 INFO] 3584 iteration USE_EMA: False, train/sup_loss: 0.0210, train/run_time: 0.0556, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:30:46,976 INFO] 3840 iteration USE_EMA: False, train/sup_loss: 0.1609, train/run_time: 0.0557, lr: 0.0000, train/prefecth_time: 0.0009 [2023-10-22 14:31:03,653 INFO] validating... [2023-10-22 14:31:05,480 INFO] confusion matrix: [[0.19 0. 0.06 0.05 0.11 0.15
  8. 0.36 0. 0.08 ] [0. 0.90625 0. 0. 0.03125 0.03125
  9. 0.03125 0. 0. ] [0. 0.01 0.53 0.14 0.03 0.
  10. 0.01 0.12 0.16 ] [0.04 0.1 0.07 0.41 0.02 0.03 0.17 0.03 0.03 0.1 ] [0.04 0.06 0. 0. 0.44 0.11 0.06 0.19 0.07 0.03 ] [0.03370787 0. 0. 0.07865169 0.02247191 0.47191011
  11. 0.38202247 0.01123596 0. ] [0. 0.06451613 0. 0.06451613 0. 0. 0.74193548 0.12903226 0. 0. ] [0.08536585 0. 0. 0. 0. 0.15853659
  12. 0.75609756 0. 0. ] [0. 0.03658537 0. 0. 0. 0.01219512
    1. 0.95121951 0. ] [0. 0.07 0.12 0. 0. 0.
    1. 0.09 0.72 ]] [2023-10-22 14:31:12,474 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:31:19,461 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/model_best.pth [2023-10-22 14:31:19,462 INFO] 4096 iteration, USE_EMA: False, train/sup_loss: 0.0226, train/run_time: 0.0535, eval/loss: 3.0230, eval/top-1-acc: 0.5674, eval/balanced_acc: 0.6117, eval/precision: 0.5731, eval/recall: 0.6117, eval/F1: 0.5634, lr: 0.0000, train/prefecth_time: 0.0011 BEST_EVAL_ACC: 0.5674, at 4096 iters [2023-10-22 14:31:36,450 INFO] 4352 iteration USE_EMA: False, train/sup_loss: 0.0025, train/run_time: 0.0576, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:31:53,131 INFO] 4608 iteration USE_EMA: False, train/sup_loss: 0.0046, train/run_time: 0.0557, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:32:09,836 INFO] 4864 iteration USE_EMA: False, train/sup_loss: 0.0062, train/run_time: 0.0558, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:32:26,536 INFO] 5120 iteration USE_EMA: False, train/sup_loss: 0.0009, train/run_time: 0.0559, lr: 0.0001, train/prefecth_time: 0.0014 [2023-10-22 14:32:43,623 INFO] 5376 iteration USE_EMA: False, train/sup_loss: 0.0009, train/run_time: 0.0557, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:33:00,300 INFO] 5632 iteration USE_EMA: False, train/sup_loss: 0.0010, train/run_time: 0.0535, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:33:17,000 INFO] 5888 iteration USE_EMA: False, train/sup_loss: 0.0055, train/run_time: 0.0557, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:33:33,726 INFO] validating... [2023-10-22 14:33:35,554 INFO] confusion matrix: [[0.43 0.25 0.09 0.01 0.05 0.05
    1. 0.01 0.11 ] [0. 0.875 0. 0. 0.03125 0.03125 0.03125 0.03125 0. 0. ] [0. 0.04 0.54 0.08 0. 0.
    1. 0.19 0.15 ] [0.05 0.17 0.08 0.38 0.02 0.02 0.15 0.04 0.03 0.06 ] [0.13 0.12 0.02 0. 0.24 0.06 0.06 0.27 0.03 0.07 ] [0.04494382 0.3258427 0.01123596 0.12359551 0. 0.47191011
    1. 0.02247191 0. ] [0. 0.06451613 0. 0.03225806 0. 0. 0.74193548 0.16129032 0. 0. ] [0.06097561 0. 0. 0. 0. 0.14634146
  18. 0.79268293 0. 0. ] [0. 0.02439024 0. 0. 0. 0.
  19. 0.02439024 0.95121951 0. ] [0. 0.15 0.08 0. 0. 0.
    1. 0.18 0.59 ]] [2023-10-22 14:33:42,682 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:33:42,683 INFO] 6144 iteration, USE_EMA: False, train/sup_loss: 0.2796, train/run_time: 0.0555, eval/loss: 3.8913, eval/top-1-acc: 0.5564, eval/balanced_acc: 0.6013, eval/precision: 0.5861, eval/recall: 0.6013, eval/F1: 0.5471, lr: 0.0000, train/prefecth_time: 0.0012 BEST_EVAL_ACC: 0.5674, at 4096 iters [2023-10-22 14:33:59,804 INFO] 6400 iteration USE_EMA: False, train/sup_loss: 0.0002, train/run_time: 0.0557, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:34:16,741 INFO] 6656 iteration USE_EMA: False, train/sup_loss: 0.0004, train/run_time: 0.0557, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:34:33,427 INFO] 6912 iteration USE_EMA: False, train/sup_loss: 2.9305, train/run_time: 0.0495, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:34:50,156 INFO] 7168 iteration USE_EMA: False, train/sup_loss: 0.0003, train/run_time: 0.0535, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:35:07,255 INFO] 7424 iteration USE_EMA: False, train/sup_loss: 0.0005, train/run_time: 0.0558, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:35:23,970 INFO] 7680 iteration USE_EMA: False, train/sup_loss: 0.0018, train/run_time: 0.0536, lr: 0.0000, train/prefecth_time: 0.0014 [2023-10-22 14:35:40,720 INFO] 7936 iteration USE_EMA: False, train/sup_loss: 0.0002, train/run_time: 0.0577, lr: 0.0000, train/prefecth_time: 0.0014 [2023-10-22 14:35:57,467 INFO] validating... [2023-10-22 14:35:59,296 INFO] confusion matrix: [[0.19 0.01 0.04 0.01 0.13 0.26
  21. 0.02 0. 0.34 ] [0.03125 0.875 0.03125 0. 0.03125 0.03125
        1. ] [0. 0.01 0.6 0.08 0.02 0.
  23. 0.02 0.14 0.13 ] [0.06 0.12 0.1 0.36 0.07 0.05 0.12 0.03 0.02 0.07 ] [0.04 0.02 0. 0. 0.49 0.04 0.04 0.29 0.01 0.07 ] [0.02247191 0.20224719 0. 0.14606742 0. 0.48314607
  24. 0.12359551 0.02247191 0. ] [0. 0.09677419 0. 0.06451613 0. 0. 0.70967742 0.12903226 0. 0. ] [0.07317073 0. 0. 0. 0.01219512 0.09756098
  25. 0.81707317 0. 0. ] [0. 0.02439024 0.01219512 0. 0. 0.01219512
    1. 0.95121951 0. ] [0. 0.09 0.18 0. 0.01 0.
    1. 0.15 0.57 ]] [2023-10-22 14:36:06,133 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:36:06,134 INFO] 8192 iteration, USE_EMA: False, train/sup_loss: 0.0001, train/run_time: 0.0575, eval/loss: 3.6727, eval/top-1-acc: 0.5625, eval/balanced_acc: 0.6046, eval/precision: 0.5584, eval/recall: 0.6046, eval/F1: 0.5546, lr: 0.0000, train/prefecth_time: 0.0010 BEST_EVAL_ACC: 0.5674, at 4096 iters [2023-10-22 14:36:13,459 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:36:14,138 INFO] Model loaded [2023-10-22 14:36:16,045 INFO] confusion matrix: [[0.71 0. 0.07 0.02 0.03 0.12
  28. 0.01 0. 0.04 ] [0.03030303 0.48484848 0.15151515 0. 0.03030303 0.
    1. 0.09090909 0.21212121] [0. 0.01 0.54 0.1 0.05 0.
    1. 0.14 0.16 ] [0.01 0.14 0.07 0.44 0.15 0. 0.05 0.03 0.04 0.07 ] [0.11 0.21 0.02 0.07 0.37 0.1 0.01 0.01 0.04 0.06 ] [0.05376344 0.01075269 0. 0.59139785 0.05376344 0.02150538
  31. 0.24731183 0.02150538 0. ] [0. 0.03125 0. 0.03125 0. 0. 0.90625 0.03125 0. 0. ] [0. 0.01041667 0. 0. 0. 0.10416667
  32. 0.875 0. 0.01041667] [0.02409639 0.04819277 0.27710843 0. 0.02409639 0.
  33. 0.02409639 0.59036145 0.01204819] [0. 0.02 0.14 0. 0.05 0.
    1. 0.13 0.66 ]] [2023-10-22 14:36:16,046 INFO] Model result - eval/best_acc : 0.5674019607843137 [2023-10-22 14:36:16,046 INFO] Model result - eval/best_it : 4095 [2023-10-22 14:36:16,046 INFO] Model result - test/best_acc : 0.5400238948626045 [2023-10-22 14:36:16,046 WARNING] GPU 0 training is FINISHED

4epoch； Semi-supervised-learning-main$ python train.py --c config/usb_audio/supervised/supervised_urbansound8k_100_0.yaml train.py:185: UserWarning: You have chosen to seed training. This will turn on the CUDNN deterministic setting, which can slow down your training considerably! You may see unexpected behavior when restarting from checkpoints. warnings.warn('You have chosen to seed training. ' [2023-10-22 14:36:31,817 INFO] Use GPU: None for training /media/ubuntu20/D/wyc/fuxian/Semi-supervised-learning-main/semilearn/datasets/utils.py:38: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. data, targets = np.array(data), np.array(targets) [2023-10-22 14:36:33,448 INFO] unlabeled data number: 7079, labeled data number 100 [2023-10-22 14:36:33,448 INFO] Create train and test data loaders [2023-10-22 14:37:13,542 INFO] [!] data loader keys: dict_keys(['train_lb', 'train_ulb', 'eval', 'test']) [2023-10-22 14:37:34,191 INFO] Create optimizer and scheduler [2023-10-22 14:37:34,193 INFO] Number of Trainable Params: 94969994 [2023-10-22 14:37:35,883 INFO] Arguments: Namespace(algorithm='supervised', amp=False, batch_size=8, c='config/usb_audio/supervised/supervised_urbansound8k_100_0.yaml', clip=0.0, clip_grad=0, crop_ratio=0.875, data_dir='./data', dataset='urbansound8k', dist_backend='nccl', dist_url='tcp://127.0.0.1:22474', distributed=False, ema_m=0.0, epoch=4, eval_batch_size=16, gpu=None, imb_algorithm=None, img_size=32, include_lb_to_ulb=True, layer_decay=0.75, lb_dest_len=100, lb_imb_ratio=1, load_path='./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth', lr=5e-05, max_length=512, max_length_seconds=4.0, momentum=0.9, multiprocessing_distributed=False, net='hubert_base', net_from_name=False, num_classes=10, num_eval_iter=2048, num_labels=100, num_log_iter=256, num_train_iter=4096, num_warmup_iter=5120, num_workers=4, optim='AdamW', overwrite=True, pretrain_path='', rank=0, resume=False, sample_rate=16000, save_dir='./saved_models/usb_audio', save_name='supervised_urbansound8k_100_0', seed=0, train_sampler='RandomSampler', ulb_dest_len=7079, ulb_imb_ratio=1, ulb_loss_ratio=1.0, ulb_num_labels=None, uratio=1, use_aim=False, use_cat=False, use_pretrain=False, use_tensorboard=True, use_wandb=False, weight_decay=2e-05, world_size=1) [2023-10-22 14:37:35,883 INFO] Resume load path ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth does not exist [2023-10-22 14:37:35,883 INFO] Model training [2023-10-22 14:37:52,823 INFO] 256 iteration USE_EMA: False, train/sup_loss: 2.2898, train/run_time: 0.0554, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:38:09,517 INFO] 512 iteration USE_EMA: False, train/sup_loss: 2.2459, train/run_time: 0.0534, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:38:26,149 INFO] 768 iteration USE_EMA: False, train/sup_loss: 2.0217, train/run_time: 0.0554, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:38:42,741 INFO] 1024 iteration USE_EMA: False, train/sup_loss: 1.6001, train/run_time: 0.0532, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:38:59,719 INFO] 1280 iteration USE_EMA: False, train/sup_loss: 1.1751, train/run_time: 0.0536, lr: 0.0000, train/prefecth_time: 0.0008 [2023-10-22 14:39:16,388 INFO] 1536 iteration USE_EMA: False, train/sup_loss: 0.6881, train/run_time: 0.0555, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:39:33,051 INFO] 1792 iteration USE_EMA: False, train/sup_loss: 0.4191, train/run_time: 0.0574, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:39:49,708 INFO] validating... [2023-10-22 14:39:51,540 INFO] confusion matrix: [[0.19 0. 0.11 0.07 0.06 0.16

  1. 0.35 0. 0.06 ] [0.15625 0.53125 0. 0.125 0.125 0.
  2. 0.0625 0. 0. ] [0. 0.04 0.59 0.09 0.06 0.
  3. 0.01 0.12 0.09 ] [0.02 0.1 0.1 0.41 0.02 0.04 0.13 0.08 0.01 0.09 ] [0.12 0.05 0. 0. 0.37 0.06 0.11 0.25 0.02 0.02 ] [0.26966292 0. 0.01123596 0.03370787 0. 0.24719101
  4. 0.42696629 0.01123596 0. ] [0. 0.12903226 0. 0.06451613 0. 0. 0.74193548 0.06451613 0. 0. ] [0.06097561 0. 0. 0. 0.01219512 0.06097561
  5. 0.86585366 0. 0. ] [0. 0.02439024 0. 0. 0.01219512 0.
  6. 0.03658537 0.92682927 0. ] [0. 0.06 0.18 0.01 0.02 0.
    1. 0.12 0.61 ]] [2023-10-22 14:39:52,458 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:39:53,347 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/model_best.pth [2023-10-22 14:39:53,348 INFO] 2048 iteration, USE_EMA: False, train/sup_loss: 0.3801, train/run_time: 0.0552, eval/loss: 1.9100, eval/top-1-acc: 0.5221, eval/balanced_acc: 0.5483, eval/precision: 0.5193, eval/recall: 0.5483, eval/F1: 0.5103, lr: 0.0000, train/prefecth_time: 0.0010 BEST_EVAL_ACC: 0.5221, at 2048 iters [2023-10-22 14:40:10,441 INFO] 2304 iteration USE_EMA: False, train/sup_loss: 0.0385, train/run_time: 0.0578, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:40:27,202 INFO] 2560 iteration USE_EMA: False, train/sup_loss: 1.6380, train/run_time: 0.0515, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:40:43,915 INFO] 2816 iteration USE_EMA: False, train/sup_loss: 0.0323, train/run_time: 0.0556, lr: 0.0000, train/prefecth_time: 0.0011 [2023-10-22 14:41:00,652 INFO] 3072 iteration USE_EMA: False, train/sup_loss: 0.6749, train/run_time: 0.0534, lr: 0.0000, train/prefecth_time: 0.0010 [2023-10-22 14:41:17,699 INFO] 3328 iteration USE_EMA: False, train/sup_loss: 0.0032, train/run_time: 0.0577, lr: 0.0000, train/prefecth_time: 0.0009 [2023-10-22 14:41:34,457 INFO] 3584 iteration USE_EMA: False, train/sup_loss: 0.0210, train/run_time: 0.0556, lr: 0.0000, train/prefecth_time: 0.0013 [2023-10-22 14:41:51,176 INFO] 3840 iteration USE_EMA: False, train/sup_loss: 0.1609, train/run_time: 0.0556, lr: 0.0000, train/prefecth_time: 0.0012 [2023-10-22 14:42:07,815 INFO] validating... [2023-10-22 14:42:09,646 INFO] confusion matrix: [[0.19 0. 0.06 0.05 0.11 0.15
  8. 0.36 0. 0.08 ] [0. 0.90625 0. 0. 0.03125 0.03125
  9. 0.03125 0. 0. ] [0. 0.01 0.53 0.14 0.03 0.
  10. 0.01 0.12 0.16 ] [0.04 0.1 0.07 0.41 0.02 0.03 0.17 0.03 0.03 0.1 ] [0.04 0.06 0. 0. 0.44 0.11 0.06 0.19 0.07 0.03 ] [0.03370787 0. 0. 0.07865169 0.02247191 0.47191011
  11. 0.38202247 0.01123596 0. ] [0. 0.06451613 0. 0.06451613 0. 0. 0.74193548 0.12903226 0. 0. ] [0.08536585 0. 0. 0. 0. 0.15853659
  12. 0.75609756 0. 0. ] [0. 0.03658537 0. 0. 0. 0.01219512
    1. 0.95121951 0. ] [0. 0.07 0.12 0. 0. 0.
    1. 0.09 0.72 ]] [2023-10-22 14:42:16,593 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:42:23,699 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/model_best.pth [2023-10-22 14:42:23,700 INFO] 4096 iteration, USE_EMA: False, train/sup_loss: 0.0226, train/run_time: 0.0534, eval/loss: 3.0230, eval/top-1-acc: 0.5674, eval/balanced_acc: 0.6117, eval/precision: 0.5731, eval/recall: 0.6117, eval/F1: 0.5634, lr: 0.0000, train/prefecth_time: 0.0010 BEST_EVAL_ACC: 0.5674, at 4096 iters [2023-10-22 14:42:30,586 INFO] model saved: ./saved_models/usb_audio/supervised_urbansound8k_100_0/latest_model.pth [2023-10-22 14:42:31,242 INFO] Model loaded [2023-10-22 14:42:33,139 INFO] confusion matrix: [[0.7 0. 0.02 0. 0.05 0.1
    1. 0.01 0.12 ] [0. 0.45454545 0.24242424 0. 0.03030303 0. 0.03030303 0. 0. 0.24242424] [0. 0. 0.42 0.14 0.03 0.
    1. 0.12 0.29 ] [0.01 0.14 0.07 0.5 0.05 0. 0.11 0.03 0.03 0.06 ] [0.03 0.17 0.01 0.07 0.37 0.18 0.04 0.06 0.02 0.05 ] [0.07526882 0. 0. 0.5483871 0.07526882 0.07526882 0.01075269 0.21505376 0. 0. ] [0. 0.03125 0. 0. 0. 0. 0.9375 0.03125 0. 0. ] [0. 0.01041667 0. 0. 0.01041667 0.23958333 0.02083333 0.71875 0. 0. ] [0.01204819 0.03614458 0.3253012 0. 0.02409639 0. 0.01204819 0.02409639 0.55421687 0.01204819] [0. 0.01 0.1 0. 0.07 0.
  17. 0.01 0.1 0.71 ]] [2023-10-22 14:42:33,140 INFO] Model result - eval/best_acc : 0.5674019607843137 [2023-10-22 14:42:33,141 INFO] Model result - eval/best_it : 4095 [2023-10-22 14:42:33,141 INFO] Model result - test/best_acc : 0.5221027479091995 [2023-10-22 14:42:33,141 WARNING] GPU 0 training is FINISHED
Hhhhhhao commented 1 year ago

This might be normal for fully supervised training on 100 labels of urbansound, where the model is learning nothing.

yichenwang231 commented 1 year ago

I compared the classifier weights of the model when epoch was set to 8.The classifier weights I tested on the validation set at the end of the fourth epoch were different from the classifier weights tested on the test set when the model loaded into the fourth epoch after training was completed.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stalled for 5 days with no activity.