NAN loss

[tzt101]

Hi, I just trained cvt13-224 model with the default settings, but got NAN loss after several epochs. Does anyone have trained this model sucessfully?

[leoxiaobin]

hi, @tzt101,

Could you paste the printed configuration for your job?

[tzt101]

This is the configuration, I just keep the default settings. AMP: ENABLED: true MEMORY_FORMAT: nchw AUG: COLOR_JITTER:

• 0.4
• 0.4
• 0.4
• 0.1
• 0.0 DROPBLOCK_BLOCK_SIZE: 7 DROPBLOCK_KEEP_PROB: 1.0 DROPBLOCK_LAYERS:
• 3
• 4 GAUSSIAN_BLUR: 0.0 GRAY_SCALE: 0.0 INTERPOLATION: 2 MIXCUT: 1.0 MIXCUT_AND_MIXUP: false MIXCUT_MINMAX: [] MIXUP: 0.8 MIXUP_MODE: batch MIXUP_PROB: 1.0 MIXUP_SWITCH_PROB: 0.5 RATIO:
• 0.75
• 1.3333333333333333 SCALE:
• 0.08
• 1.0 TIMM_AUG: AUTO_AUGMENT: rand-m9-mstd0.5-inc1 COLOR_JITTER: 0.4 HFLIP: 0.5 INTERPOLATION: bicubic RE_COUNT: 1 RE_MODE: pixel RE_PROB: 0.25 RE_SPLIT: false USE_LOADER: true USE_TRANSFORM: false VFLIP: 0.0 BASE:
  • '' CUDNN: BENCHMARK: true DETERMINISTIC: false ENABLED: true DATASET: DATASET: imagenet DATA_FORMAT: jpg LABELMAP: '' ROOT: /home/tzt/dataset/imagenet/ SAMPLER: default TARGET_SIZE: -1 TEST_SET: val TEST_TSV_LIST: [] TRAIN_SET: train TRAIN_TSV_LIST: [] DATA_DIR: '' DEBUG: DEBUG: false DIST_BACKEND: nccl FINETUNE: BASE_LR: 0.003 BATCH_SIZE: 512 EVAL_EVERY: 3000 FINETUNE: false FROZEN_LAYERS: [] LR_SCHEDULER: DECAY_TYPE: step TRAIN_MODE: true USE_TRAIN_AUG: false GPUS:
  • 0 INPUT: MEAN:
• 0.485
• 0.456
• 0.406 STD:
• 0.229
• 0.224
• 0.225 LOSS: LABEL_SMOOTHING: 0.1 LOSS: softmax MODEL: INIT_WEIGHTS: true NAME: cls_cvt NUM_CLASSES: 1000 PRETRAINED: '' PRETRAINED_LAYERS:
• '*' SPEC: ATTN_DROP_RATE:
  • 0.0
  • 0.0
  • 0.0 CLS_TOKEN:
  • false
  • false
  • true DEPTH:
  • 1
  • 2
  • 10 DIM_EMBED:
  • 64
  • 192
  • 384 DROP_PATH_RATE:
  • 0.0
  • 0.0
  • 0.1 DROP_RATE:
  • 0.0
  • 0.0
  • 0.0 INIT: trunc_norm KERNEL_QKV:
  • 3
  • 3
  • 3 MLP_RATIO:
  • 4.0
  • 4.0
  • 4.0 NUM_HEADS:
  • 1
  • 3
  • 6 NUM_STAGES: 3 PADDING_KV:
  • 1
  • 1
  • 1 PADDING_Q:
  • 1
  • 1
  • 1 PATCH_PADDING:
  • 2
  • 1
  • 1 PATCH_SIZE:
  • 7
  • 3
  • 3 PATCH_STRIDE:
  • 4
  • 2
  • 2 POS_EMBED:
  • false
  • false
  • false QKV_BIAS:
  • true
  • true
  • true QKV_PROJ_METHOD:
  • dw_bn
  • dw_bn
  • dw_bn STRIDE_KV:
  • 2
  • 2
  • 2 STRIDE_Q:
  • 1
  • 1
  • 1 MODEL_SUMMARY: false MULTIPROCESSING_DISTRIBUTED: true NAME: cvt-13-224x224 OUTPUT_DIR: OUTPUT/ PIN_MEMORY: true PRINT_FREQ: 500 RANK: 0 TEST: BATCH_SIZE_PER_GPU: 32 CENTER_CROP: true IMAGE_SIZE:
• 224
• 224 INTERPOLATION: 3 MODEL_FILE: '' REAL_LABELS: false VALID_LABELS: '' TRAIN: AUTO_RESUME: true BATCH_SIZE_PER_GPU: 128 BEGIN_EPOCH: 0 CHECKPOINT: '' CLIP_GRAD_NORM: 0.0 DETECT_ANOMALY: false END_EPOCH: 300 EVAL_BEGIN_EPOCH: 0 GAMMA1: 0.99 GAMMA2: 0.0 IMAGE_SIZE:
• 224
• 224 LR: 0.16 LR_SCHEDULER: ARGS: cooldown_epochs: 10 decay_rate: 0.1 epochs: 300 min_lr: 1.0e-05 sched: cosine warmup_epochs: 5 warmup_lr: 1.0e-06 METHOD: timm MOMENTUM: 0.9 NESTEROV: true OPTIMIZER: adamW OPTIMIZER_ARGS: {} SAVE_ALL_MODELS: false SCALE_LR: true SHUFFLE: true WD: 0.05 WITHOUT_WD_LIST:
• bn
• bias
• ln VERBOSE: true WORKERS: 6

[leoxiaobin]

it seems that you are using a larger LR. If you specify BATCH_SIZE_PER_GPU to 128, you should specify LR to 0.000125. The LR in our config is with respect to BATCH_SIZE_PER_GPU. You are using a much larger LR than our original config. I guess that's the reason you got NaN error.

[tzt101]

Thank you very much! I will try to use small lr later.