search

MILVLG / openvqa

A lightweight, scalable, and general framework for visual question answering research
Apache License 2.0
318 stars 64 forks source link

Reproducing model zoo results on GQA #61

Closed mrfarazi closed 4 years ago

mrfarazi commented 4 years ago

Hello, I am trying to reproduce the results reported on model zoo for GQA dataset. I am doing Train+val -> Test-dev with mcan_large on GQA dataset with the following code:

python3 run.py --RUN='train' --SPLIT='train+val' --MODEL='mcan_large' --DATASET='gqa' --GPU='7' --VERSION=''default_frcn+bbox+grid

The accuracy on local evaluation is 56.23% and from GQA evaluation server is 56.57% (with in reasonable limit I guess). But the reported accuracy in model zoo for MCAN-large (frcn+bbox+grid) is 58.10%. I think this is a significant difference in accuracy. Could you please tell me if I am doing something wrong? I have used all provided features as is, and did not modify the code.

nbgao commented 4 years ago

If possible, you could paste log information here.

mrfarazi commented 4 years ago

@nbgao The log with first and last epoch

{ BATCH_SIZE        }->64
{ BBOXFEAT_EMB_SIZE }->2048
{ CACHE_PATH        }->./results/cache
{ CKPTS_PATH        }->./ckpts
{ CKPT_EPOCH        }->0
{ CKPT_PATH         }->None
{ CKPT_VERSION      }->4120150
{ DATASET           }->gqa
{ DATA_PATH         }->{'vqa': './data/vqa', 'clevr': './data/clevr', 'gqa': './data/gqa'}
{ DATA_ROOT         }->./data
{ DEVICES           }->[0]
{ DROPOUT_R         }->0.1
{ EVAL_BATCH_SIZE   }->8
{ EVAL_EVERY_EPOCH  }->True
{ FEATS_PATH        }->{'vqa': {'val': './data/vqa/feats/val2014', 'train': './data/vqa/feats/train2014', 'test': './data/vqa/feats/test2015'}, 'clevr': {'val': './data/clevr/feats/val', 'train': './data/clevr/feats/train', 'test': './data/clevr/feats/test'}, 'gqa': {'default-frcn': './data/gqa/feats/gqa-frcn', 'default-grid': './data/gqa/feats/gqa-grid'}}
{ FEAT_SIZE         }->{'vqa': {'BBOX_FEAT_SIZE': (100, 5), 'FRCN_FEAT_SIZE': (100, 2048)}, 'clevr': {'GRID_FEAT_SIZE': (196, 1024)}, 'gqa': {'GRID_FEAT_SIZE': (49, 2048), 'BBOX_FEAT_SIZE': (100, 5), 'FRCN_FEAT_SIZE': (100, 2048)}}
{ FF_SIZE           }->4096
{ FLAT_GLIMPSES     }->1
{ FLAT_MLP_SIZE     }->512
{ FLAT_OUT_SIZE     }->2048
{ GPU               }->7
{ GRAD_ACCU_STEPS   }->4
{ GRAD_NORM_CLIP    }->-1
{ HIDDEN_SIZE       }->1024
{ LAYER             }->6
{ LOG_PATH          }->./results/log
{ LOSS_FUNC         }->ce
{ LOSS_FUNC_NAME_DICT }->{'bce': 'BCEWithLogitsLoss', 'kld': 'KLDivLoss', 'ce': 'CrossEntropyLoss', 'mse': 'MSELoss'}
{ LOSS_FUNC_NONLINEAR }->{'bce': [None, None], 'kld': ['log_softmax', None], 'ce': [None, 'flat'], 'mse': [None, None]}
{ LOSS_REDUCTION    }->sum
{ LR_BASE           }->5e-05
{ LR_DECAY_LIST     }->[8, 10]
{ LR_DECAY_R        }->0.2
{ MAX_EPOCH         }->11
{ MODEL             }->mcan_large
{ MODEL_USE         }->mcan
{ MULTI_HEAD        }->8
{ NUM_WORKERS       }->8
{ N_GPU             }->1
{ OPT               }->Adam
{ OPT_PARAMS        }->{'betas': (0.9, 0.98), 'weight_decay': 0, 'eps': 1e-09, 'amsgrad': False}
{ PIN_MEM           }->True
{ PRED_PATH         }->./results/pred
{ RAW_PATH          }->{'vqa': {'val-anno': './data/vqa/raw/v2_mscoco_val2014_annotations.json', 'val': './data/vqa/raw/v2_OpenEnded_mscoco_val2014_questions.json', 'test': './data/vqa/raw/v2_OpenEnded_mscoco_test2015_questions.json', 'vg': './data/vqa/raw/VG_questions.json', 'train-anno': './data/vqa/raw/v2_mscoco_train2014_annotations.json', 'train': './data/vqa/raw/v2_OpenEnded_mscoco_train2014_questions.json', 'vg-anno': './data/vqa/raw/VG_annotations.json'}, 'clevr': {'val': './data/clevr/raw/questions/CLEVR_val_questions.json', 'train': './data/clevr/raw/questions/CLEVR_train_questions.json', 'test': './data/clevr/raw/questions/CLEVR_test_questions.json'}, 'gqa': {'val': './data/gqa/raw/questions1.2/val_balanced_questions.json', 'val_all': './data/gqa/raw/questions1.2/val_all_questions.json', 'test': './data/gqa/raw/questions1.2/submission_all_questions.json', 'testdev_all': './data/gqa/raw/questions1.2/testdev_all_questions.json', 'val_choices': './data/gqa/raw/eval/val_choices.json', 'train_choices': './data/gqa/raw/eval/train_choices', 'train': './data/gqa/raw/questions1.2/train_balanced_questions.json', 'testdev': './data/gqa/raw/questions1.2/testdev_balanced_questions.json'}}
{ RESULT_PATH       }->./results/result_test
{ RESUME            }->False
{ RUN_MODE          }->train
{ SEED              }->4120150
{ SPLIT             }->{'val': 'testdev', 'train': 'train+val', 'test': 'test'}
{ SPLITS            }->{'vqa': {'val': 'val', 'train': '', 'test': 'test'}, 'clevr': {'val': 'val', 'train': '', 'test': 'test'}, 'gqa': {'val': 'testdev', 'train': 'train+val', 'test': 'test'}}
{ SUB_BATCH_SIZE    }->16
{ TASK_LOSS_CHECK   }->{'vqa': ['bce', 'kld'], 'clevr': ['ce'], 'gqa': ['ce']}
{ TEST_SAVE_PRED    }->False
{ TRAIN_SPLIT       }->train+val
{ USE_AUX_FEAT      }->True
{ USE_BBOX_FEAT     }->True
{ USE_GLOVE         }->True
{ VERBOSE           }->True
{ VERSION           }->default_grid+bb+frcn
{ WARMUP_EPOCH      }->2
{ WORD_EMBED_SIZE   }->300
=====================================
nowTime: 2020-01-15 17:17:00
Epoch: 1, Loss: 1.68896448995679, Lr: 3.3333333333333335e-05
Elapsed time: 12386, Speed(s/batch): 0.7374428591009378

Binary: 56.75%
Open: 34.61%
Accuracy: 44.77%
Distribution: 4.05 (lower is better)
Accuracy / structural type:
  choose: 60.76% (1129 questions)
  compare: 57.05% (589 questions)
  logical: 57.35% (1803 questions)
  query: 34.61% (6805 questions)
  verify: 54.17% (2252 questions)
Accuracy / semantic type:
  attr: 48.98% (5186 questions)
  cat: 40.38% (1149 questions)
  global: 49.68% (157 questions)
  obj: 68.12% (778 questions)
  rel: 38.04% (5308 questions)
Accuracy / steps number:
  1: 60.34% (237 questions)
  2: 40.72% (6395 questions)
  3: 45.99% (4266 questions)
  4: 47.04% (793 questions)
  5: 60.46% (822 questions)
  6: 68.29% (41 questions)
  7: 100.00% (20 questions)
  8: 100.00% (3 questions)
  9: 100.00% (1 questions)
Accuracy / words number:
  3: 23.18% (151 questions)
  4: 41.75% (630 questions)
  5: 35.04% (1290 questions)
  6: 40.98% (2074 questions)
  7: 42.02% (1642 questions)
  8: 46.33% (1185 questions)
  9: 49.65% (1281 questions)
  10: 50.92% (1249 questions)
  11: 46.98% (994 questions)
  12: 52.04% (638 questions)
  13: 49.78% (462 questions)
  14: 47.25% (345 questions)
  15: 51.90% (237 questions)
  16: 51.28% (117 questions)
  17: 45.74% (94 questions)
  18: 53.95% (76 questions)
  19: 58.14% (43 questions)
  20: 56.25% (32 questions)
  21: 42.11% (19 questions)
  22: 50.00% (12 questions)
  23: 25.00% (4 questions)
  24: 100.00% (2 questions)
  25: 100.00% (1 questions)

nowTime: 2020-01-17 03:49:53
Epoch: 11, Loss: 0.27490622613804777, Lr: 2.0000000000000003e-06
Elapsed time: 12362, Speed(s/batch): 0.7360429190743336

Binary: 75.13%
Open: 40.21%
Accuracy: 56.23%
Distribution: 1.93 (lower is better)
Accuracy / structural type:
  choose: 77.06% (1129 questions)
  compare: 62.48% (589 questions)
  logical: 73.60% (1803 questions)
  query: 40.21% (6805 questions)
  verify: 78.69% (2252 questions)
Accuracy / semantic type:
  attr: 64.38% (5186 questions)
  cat: 43.34% (1149 questions)
  global: 53.50% (157 questions)
  obj: 84.58% (778 questions)
  rel: 46.99% (5308 questions)
Accuracy / steps number:
  1: 70.46% (237 questions)
  2: 49.84% (6395 questions)
  3: 59.94% (4266 questions)
  4: 66.20% (793 questions)
  5: 70.56% (822 questions)
  6: 80.49% (41 questions)
  7: 100.00% (20 questions)
  8: 100.00% (3 questions)
  9: 100.00% (1 questions)
Accuracy / words number:
  3: 31.13% (151 questions)
  4: 50.32% (630 questions)
  5: 43.33% (1290 questions)
  6: 52.56% (2074 questions)
  7: 56.70% (1642 questions)
  8: 57.81% (1185 questions)
  9: 62.61% (1281 questions)
  10: 62.21% (1249 questions)
  11: 56.64% (994 questions)
  12: 60.97% (638 questions)
  13: 57.58% (462 questions)
  14: 67.25% (345 questions)
  15: 64.98% (237 questions)
  16: 66.67% (117 questions)
  17: 59.57% (94 questions)
  18: 68.42% (76 questions)
  19: 76.74% (43 questions)
  20: 53.12% (32 questions)
  21: 63.16% (19 questions)
  22: 75.00% (12 questions)
  23: 25.00% (4 questions)
  24: 100.00% (2 questions)
  25: 100.00% (1 questions)```
nbgao commented 4 years ago

@mrfarazi We set the SEED=1018 in this reported mcan_large model. The training log file is as follow.

{ BATCH_SIZE }->64 { BBOXFEAT_EMB_SIZE }->2048 { CACHE_PATH }->./results/cache { CKPTS_PATH }->./ckpts { CKPT_EPOCH }->0 { CKPT_PATH }->None { CKPT_VERSION }->4864225 { DATASET }->gqa { DATA_PATH }->{'vqa': '/data-ssd/gaopb/datasets/vqa', 'gqa': '/data-ssd/gaopb/datasets/gqa', 'clevr': '/data-ssd/gaopb/datasets/clevr'} { DATA_ROOT }->/data-ssd/gaopb/datasets { DEVICES }->[0] { DROPOUT_R }->0.1 { EVAL_BATCH_SIZE }->8 { EVAL_EVERY_EPOCH }->True { FEATS_PATH }->{'vqa': {'train': '/data-ssd/gaopb/datasets/vqa/feats/train2014', 'val': '/data-ssd/gaopb/datasets/vqa/feats/val2014', 'test': '/data-ssd/gaopb/datasets/vqa/feats/test2015'}, 'gqa': {'default-frcn': '/data-ssd/gaopb/datasets/gqa/feats/gqa-frcn', 'default-grid': '/data-ssd/gaopb/datasets/gqa/feats/gqa-grid'}, 'clevr': {'train': '/data-ssd/gaopb/datasets/clevr/feats/train', 'val': '/data-ssd/gaopb/datasets/clevr/feats/val', 'test': '/data-ssd/gaopb/datasets/clevr/feats/test'}} { FEAT_SIZE }->{'vqa': {'FRCN_FEAT_SIZE': 2048, 'BBOX_FEAT_SIZE': 5}, 'gqa': {'FRCN_FEAT_SIZE': 2048, 'GRID_FEAT_SIZE': 2048, 'BBOX_FEAT_SIZE': 5}, 'clevr': {'GRID_FEAT_SIZE': 1024}} { FF_SIZE }->4096 { FLAT_GLIMPSES }->1 { FLAT_MLP_SIZE }->512 { FLAT_OUT_SIZE }->2048 { GPU }->1 { GRAD_ACCU_STEPS }->4 { GRAD_NORM_CLIP }->-1 { HIDDEN_SIZE }->1024 { LAYER }->6 { LOG_PATH }->./results/log { LOSS_FUNC }->ce { LOSS_FUNC_NAME_DICT }->{'ce': 'CrossEntropyLoss', 'bce': 'BCEWithLogitsLoss', 'kld': 'KLDivLoss', 'mse': 'MSELoss'} { LOSS_FUNC_NONLINEAR }->{'ce': [None, 'flat'], 'bce': [None, None], 'kld': ['log_softmax', None], 'mse': [None, None]} { LOSS_REDUCTION }->sum { LR_BASE }->5e-05 { LR_DECAY_LIST }->[8, 10] { LR_DECAY_R }->0.2 { MAX_EPOCH }->11 { MODEL }->mcan_large { MODEL_USE }->mcan { MULTI_HEAD }->8 { NUM_WORKERS }->8 { N_GPU }->1 { OPT }->Adam { OPT_PARAMS }->{'betas': (0.9, 0.98), 'eps': 1e-09, 'weight_decay': 0, 'amsgrad': False} { PIN_MEM }->True { PRED_PATH }->./results/pred { RAW_PATH }->{'vqa': {'train': '/data-ssd/gaopb/datasets/vqa/raw/v2_OpenEnded_mscoco_train2014_questions.json', 'train-anno': '/data-ssd/gaopb/datasets/vqa/raw/v2_mscoco_train2014_annotations.json', 'val': '/data-ssd/gaopb/datasets/vqa/raw/v2_OpenEnded_mscoco_val2014_questions.json', 'val-anno': '/data-ssd/gaopb/datasets/vqa/raw/v2_mscoco_val2014_annotations.json', 'vg': '/data-ssd/gaopb/datasets/vqa/raw/VG_questions.json', 'vg-anno': '/data-ssd/gaopb/datasets/vqa/raw/VG_annotations.json', 'test': '/data-ssd/gaopb/datasets/vqa/raw/v2_OpenEnded_mscoco_test2015_questions.json'}, 'gqa': {'train': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/train_balanced_questions.json', 'val': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/val_balanced_questions.json', 'testdev': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/testdev_balanced_questions.json', 'test': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/submission_all_questions.json', 'val_all': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/val_all_questions.json', 'testdev_all': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/testdev_all_questions.json', 'train_choices': '/data-ssd/gaopb/datasets/gqa/raw/eval/train_choices', 'val_choices': '/data-ssd/gaopb/datasets/gqa/raw/eval/val_choices.json'}, 'clevr': {'train': '/data-ssd/gaopb/datasets/clevr/raw/questions/CLEVR_train_questions.json', 'val': '/data-ssd/gaopb/datasets/clevr/raw/questions/CLEVR_val_questions.json', 'test': '/data-ssd/gaopb/datasets/clevr/raw/questions/CLEVR_test_questions.json'}} { RESULT_PATH }->./results/result_test { RESUME }->False { RUN_MODE }->train { SEED }->1018 { SPLIT }->{'train': 'train+val', 'val': 'testdev', 'test': 'test'} { SPLITS }->{'vqa': {'train': '', 'val': 'val', 'test': 'test'}, 'gqa': {'train': 'train+val', 'val': 'testdev', 'test': 'test'}, 'clevr': {'train': '', 'val': 'val', 'test': 'test'}} { SUB_BATCH_SIZE }->16 { TASK_LOSS_CHECK }->{'vqa': ['bce', 'kld'], 'gqa': ['ce'], 'clevr': ['ce']} { TEST_SAVE_PRED }->False { TRAIN_SPLIT }->train+val { USE_AUX_FEAT }->True { USE_BBOX_FEAT }->True { USE_GLOVE }->True { VERBOSE }->True { VERSION }->gqa_mcan_large_train_tv1 { WARMUP_EPOCH }->2 { WORD_EMBED_SIZE }->300

nowTime: 2019-10-19 21:16:34 Epoch: 1, Loss: 1.6641010988449478, Lr: 3.3333333333333335e-05 Elapsed time: 21938, Speed(s/batch): 1.306167084564905

Binary: 58.48% Open: 34.65% Accuracy: 45.59% Distribution: 4.15 (lower is better) Accuracy / structural type: choose: 60.85% (1129 questions) compare: 44.48% (589 questions) logical: 57.90% (1803 questions) query: 34.65% (6805 questions) verify: 61.41% (2252 questions) Accuracy / semantic type: attr: 50.10% (5186 questions) cat: 36.99% (1149 questions) global: 44.59% (157 questions) obj: 68.51% (778 questions) rel: 39.71% (5308 questions) Accuracy / steps number: 1: 55.27% (237 questions) 2: 41.92% (6395 questions) 3: 46.98% (4266 questions) 4: 48.93% (793 questions) 5: 57.79% (822 questions) 6: 75.61% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 27.15% (151 questions) 4: 40.79% (630 questions) 5: 35.12% (1290 questions) 6: 41.71% (2074 questions) 7: 43.18% (1642 questions) 8: 46.50% (1185 questions) 9: 50.82% (1281 questions) 10: 50.36% (1249 questions) 11: 50.40% (994 questions) 12: 49.37% (638 questions) 13: 51.95% (462 questions) 14: 53.62% (345 questions) 15: 56.54% (237 questions) 16: 54.70% (117 questions) 17: 41.49% (94 questions) 18: 50.00% (76 questions) 19: 65.12% (43 questions) 20: 43.75% (32 questions) 21: 42.11% (19 questions) 22: 58.33% (12 questions) 23: 50.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-20 03:23:54 Epoch: 2, Loss: 1.3524777466714077, Lr: 5e-05 Elapsed time: 21912, Speed(s/batch): 1.3046081712712103

Binary: 64.06% Open: 38.65% Accuracy: 50.31% Distribution: 2.05 (lower is better) Accuracy / structural type: choose: 67.49% (1129 questions) compare: 42.44% (589 questions) logical: 60.73% (1803 questions) query: 38.65% (6805 questions) verify: 70.65% (2252 questions) Accuracy / semantic type: attr: 54.47% (5186 questions) cat: 44.39% (1149 questions) global: 48.41% (157 questions) obj: 76.74% (778 questions) rel: 43.71% (5308 questions) Accuracy / steps number: 1: 66.24% (237 questions) 2: 46.29% (6395 questions) 3: 53.19% (4266 questions) 4: 49.94% (793 questions) 5: 59.61% (822 questions) 6: 78.05% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 33.11% (151 questions) 4: 45.87% (630 questions) 5: 39.30% (1290 questions) 6: 46.72% (2074 questions) 7: 48.48% (1642 questions) 8: 53.50% (1185 questions) 9: 56.83% (1281 questions) 10: 55.24% (1249 questions) 11: 50.40% (994 questions) 12: 53.29% (638 questions) 13: 53.46% (462 questions) 14: 58.26% (345 questions) 15: 62.45% (237 questions) 16: 60.68% (117 questions) 17: 54.26% (94 questions) 18: 46.05% (76 questions) 19: 79.07% (43 questions) 20: 46.88% (32 questions) 21: 52.63% (19 questions) 22: 58.33% (12 questions) 23: 50.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-20 09:30:46 Epoch: 3, Loss: 1.1937491521311219, Lr: 5e-05 Elapsed time: 20965, Speed(s/batch): 1.2482277418097079

Binary: 70.03% Open: 37.91% Accuracy: 52.66% Distribution: 2.33 (lower is better) Accuracy / structural type: choose: 68.20% (1129 questions) compare: 52.80% (589 questions) logical: 69.61% (1803 questions) query: 37.91% (6805 questions) verify: 75.80% (2252 questions) Accuracy / semantic type: attr: 59.10% (5186 questions) cat: 42.65% (1149 questions) global: 54.14% (157 questions) obj: 82.39% (778 questions) rel: 44.12% (5308 questions) Accuracy / steps number: 1: 66.24% (237 questions) 2: 46.97% (6395 questions) 3: 55.34% (4266 questions) 4: 61.16% (793 questions) 5: 68.37% (822 questions) 6: 73.17% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 31.13% (151 questions) 4: 47.62% (630 questions) 5: 39.69% (1290 questions) 6: 49.32% (2074 questions) 7: 52.92% (1642 questions) 8: 56.12% (1185 questions) 9: 57.38% (1281 questions) 10: 58.69% (1249 questions) 11: 53.02% (994 questions) 12: 58.62% (638 questions) 13: 55.41% (462 questions) 14: 57.68% (345 questions) 15: 60.34% (237 questions) 16: 61.54% (117 questions) 17: 56.38% (94 questions) 18: 57.89% (76 questions) 19: 74.42% (43 questions) 20: 59.38% (32 questions) 21: 47.37% (19 questions) 22: 58.33% (12 questions) 23: 25.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-20 15:21:50 Epoch: 4, Loss: 1.0559891983221752, Lr: 5e-05 Elapsed time: 21643, Speed(s/batch): 1.2885841661108253

Binary: 71.19% Open: 37.90% Accuracy: 53.18% Distribution: 1.75 (lower is better) Accuracy / structural type: choose: 70.59% (1129 questions) compare: 60.78% (589 questions) logical: 70.83% (1803 questions) query: 37.90% (6805 questions) verify: 74.51% (2252 questions) Accuracy / semantic type: attr: 59.27% (5186 questions) cat: 40.73% (1149 questions) global: 56.05% (157 questions) obj: 83.03% (778 questions) rel: 45.46% (5308 questions) Accuracy / steps number: 1: 66.67% (237 questions) 2: 47.08% (6395 questions) 3: 56.47% (4266 questions) 4: 60.66% (793 questions) 5: 69.71% (822 questions) 6: 82.93% (41 questions) 7: 95.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 35.10% (151 questions) 4: 46.83% (630 questions) 5: 40.78% (1290 questions) 6: 48.70% (2074 questions) 7: 52.92% (1642 questions) 8: 56.03% (1185 questions) 9: 59.64% (1281 questions) 10: 57.65% (1249 questions) 11: 55.53% (994 questions) 12: 59.87% (638 questions) 13: 56.49% (462 questions) 14: 60.00% (345 questions) 15: 65.82% (237 questions) 16: 58.12% (117 questions) 17: 50.00% (94 questions) 18: 56.58% (76 questions) 19: 72.09% (43 questions) 20: 56.25% (32 questions) 21: 57.89% (19 questions) 22: 58.33% (12 questions) 23: 50.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-20 21:24:14 Epoch: 5, Loss: 0.9538466596906512, Lr: 5e-05 Elapsed time: 21822, Speed(s/batch): 1.29927454803296

Binary: 72.30% Open: 38.91% Accuracy: 54.24% Distribution: 2.00 (lower is better) Accuracy / structural type: choose: 71.57% (1129 questions) compare: 59.76% (589 questions) logical: 71.66% (1803 questions) query: 38.91% (6805 questions) verify: 76.47% (2252 questions) Accuracy / semantic type: attr: 60.59% (5186 questions) cat: 44.04% (1149 questions) global: 51.59% (157 questions) obj: 83.42% (778 questions) rel: 46.04% (5308 questions) Accuracy / steps number: 1: 64.98% (237 questions) 2: 48.30% (6395 questions) 3: 57.34% (4266 questions) 4: 62.55% (793 questions) 5: 70.56% (822 questions) 6: 80.49% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 34.44% (151 questions) 4: 46.19% (630 questions) 5: 40.70% (1290 questions) 6: 52.07% (2074 questions) 7: 52.62% (1642 questions) 8: 56.79% (1185 questions) 9: 60.27% (1281 questions) 10: 60.29% (1249 questions) 11: 55.73% (994 questions) 12: 59.40% (638 questions) 13: 57.58% (462 questions) 14: 59.13% (345 questions) 15: 63.71% (237 questions) 16: 62.39% (117 questions) 17: 60.64% (94 questions) 18: 65.79% (76 questions) 19: 79.07% (43 questions) 20: 62.50% (32 questions) 21: 68.42% (19 questions) 22: 58.33% (12 questions) 23: 25.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-21 03:29:42 Epoch: 6, Loss: 0.8768809864153777, Lr: 5e-05 Elapsed time: 21838, Speed(s/batch): 1.3002370217494323

Binary: 74.09% Open: 38.41% Accuracy: 54.79% Distribution: 2.15 (lower is better) Accuracy / structural type: choose: 73.34% (1129 questions) compare: 64.52% (589 questions) logical: 72.60% (1803 questions) query: 38.41% (6805 questions) verify: 78.15% (2252 questions) Accuracy / semantic type: attr: 61.76% (5186 questions) cat: 43.17% (1149 questions) global: 58.60% (157 questions) obj: 83.03% (778 questions) rel: 46.23% (5308 questions) Accuracy / steps number: 1: 69.20% (237 questions) 2: 48.58% (6395 questions) 3: 58.02% (4266 questions) 4: 62.42% (793 questions) 5: 72.26% (822 questions) 6: 80.49% (41 questions) 7: 95.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 27.81% (151 questions) 4: 50.48% (630 questions) 5: 40.62% (1290 questions) 6: 51.25% (2074 questions) 7: 53.59% (1642 questions) 8: 56.03% (1185 questions) 9: 60.66% (1281 questions) 10: 62.61% (1249 questions) 11: 55.13% (994 questions) 12: 60.97% (638 questions) 13: 59.09% (462 questions) 14: 64.93% (345 questions) 15: 65.82% (237 questions) 16: 63.25% (117 questions) 17: 59.57% (94 questions) 18: 61.84% (76 questions) 19: 76.74% (43 questions) 20: 59.38% (32 questions) 21: 52.63% (19 questions) 22: 66.67% (12 questions) 23: 25.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-21 09:35:25 Epoch: 7, Loss: 0.8124484499360088, Lr: 5e-05 Elapsed time: 21844, Speed(s/batch): 1.3006059032134938

Binary: 74.33% Open: 39.71% Accuracy: 55.60% Distribution: 2.03 (lower is better) Accuracy / structural type: choose: 74.14% (1129 questions) compare: 62.31% (589 questions) logical: 72.43% (1803 questions) query: 39.71% (6805 questions) verify: 79.09% (2252 questions) Accuracy / semantic type: attr: 62.26% (5186 questions) cat: 45.08% (1149 questions) global: 56.05% (157 questions) obj: 83.93% (778 questions) rel: 47.19% (5308 questions) Accuracy / steps number: 1: 73.00% (237 questions) 2: 49.73% (6395 questions) 3: 58.79% (4266 questions) 4: 61.92% (793 questions) 5: 71.05% (822 questions) 6: 80.49% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 32.45% (151 questions) 4: 50.63% (630 questions) 5: 41.09% (1290 questions) 6: 52.36% (2074 questions) 7: 55.66% (1642 questions) 8: 56.12% (1185 questions) 9: 61.67% (1281 questions) 10: 63.97% (1249 questions) 11: 57.85% (994 questions) 12: 59.09% (638 questions) 13: 56.71% (462 questions) 14: 64.35% (345 questions) 15: 61.60% (237 questions) 16: 67.52% (117 questions) 17: 55.32% (94 questions) 18: 67.11% (76 questions) 19: 74.42% (43 questions) 20: 59.38% (32 questions) 21: 68.42% (19 questions) 22: 75.00% (12 questions) 23: 25.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-21 15:41:13 Epoch: 8, Loss: 0.760144480078704, Lr: 5e-05 Elapsed time: 22146, Speed(s/batch): 1.3185374425916225

Binary: 74.61% Open: 39.66% Accuracy: 55.70% Distribution: 1.83 (lower is better) Accuracy / structural type: choose: 73.69% (1129 questions) compare: 65.70% (589 questions) logical: 73.43% (1803 questions) query: 39.66% (6805 questions) verify: 78.33% (2252 questions) Accuracy / semantic type: attr: 62.86% (5186 questions) cat: 44.82% (1149 questions) global: 59.87% (157 questions) obj: 85.09% (778 questions) rel: 46.63% (5308 questions) Accuracy / steps number: 1: 68.35% (237 questions) 2: 49.27% (6395 questions) 3: 59.52% (4266 questions) 4: 62.67% (793 questions) 5: 73.60% (822 questions) 6: 68.29% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 32.45% (151 questions) 4: 48.10% (630 questions) 5: 42.40% (1290 questions) 6: 51.35% (2074 questions) 7: 54.63% (1642 questions) 8: 58.23% (1185 questions) 9: 62.53% (1281 questions) 10: 62.77% (1249 questions) 11: 57.65% (994 questions) 12: 59.72% (638 questions) 13: 59.96% (462 questions) 14: 64.06% (345 questions) 15: 67.93% (237 questions) 16: 61.54% (117 questions) 17: 61.70% (94 questions) 18: 67.11% (76 questions) 19: 76.74% (43 questions) 20: 56.25% (32 questions) 21: 63.16% (19 questions) 22: 66.67% (12 questions) 23: 50.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-21 21:52:02 Epoch: 9, Loss: 0.4970882366093684, Lr: 1e-05 Elapsed time: 22065, Speed(s/batch): 1.3137314969597897

Binary: 77.33% Open: 41.25% Accuracy: 57.81% Distribution: 1.66 (lower is better) Accuracy / structural type: choose: 77.41% (1129 questions) compare: 66.38% (589 questions) logical: 76.04% (1803 questions) query: 41.25% (6805 questions) verify: 81.17% (2252 questions) Accuracy / semantic type: attr: 65.43% (5186 questions) cat: 46.04% (1149 questions) global: 56.05% (157 questions) obj: 86.89% (778 questions) rel: 48.70% (5308 questions) Accuracy / steps number: 1: 71.73% (237 questions) 2: 51.23% (6395 questions) 3: 61.44% (4266 questions) 4: 66.20% (793 questions) 5: 75.79% (822 questions) 6: 78.05% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 31.79% (151 questions) 4: 49.21% (630 questions) 5: 44.73% (1290 questions) 6: 54.97% (2074 questions) 7: 56.64% (1642 questions) 8: 60.00% (1185 questions) 9: 63.08% (1281 questions) 10: 65.57% (1249 questions) 11: 60.06% (994 questions) 12: 61.91% (638 questions) 13: 60.39% (462 questions) 14: 66.96% (345 questions) 15: 65.40% (237 questions) 16: 68.38% (117 questions) 17: 63.83% (94 questions) 18: 71.05% (76 questions) 19: 81.40% (43 questions) 20: 53.12% (32 questions) 21: 63.16% (19 questions) 22: 66.67% (12 questions) 23: 50.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-22 04:01:29 Epoch: 10, Loss: 0.3919823110563895, Lr: 1e-05 Elapsed time: 21820, Speed(s/batch): 1.2991480892628366

Binary: 77.48% Open: 40.60% Accuracy: 57.53% Distribution: 1.50 (lower is better) Accuracy / structural type: choose: 77.77% (1129 questions) compare: 65.03% (589 questions) logical: 77.04% (1803 questions) query: 40.60% (6805 questions) verify: 80.95% (2252 questions) Accuracy / semantic type: attr: 65.16% (5186 questions) cat: 45.69% (1149 questions) global: 54.14% (157 questions) obj: 87.02% (778 questions) rel: 48.42% (5308 questions) Accuracy / steps number: 1: 71.73% (237 questions) 2: 50.51% (6395 questions) 3: 61.20% (4266 questions) 4: 66.96% (793 questions) 5: 77.62% (822 questions) 6: 78.05% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 30.46% (151 questions) 4: 50.16% (630 questions) 5: 43.33% (1290 questions) 6: 53.76% (2074 questions) 7: 56.76% (1642 questions) 8: 59.66% (1185 questions) 9: 64.01% (1281 questions) 10: 64.13% (1249 questions) 11: 59.76% (994 questions) 12: 63.01% (638 questions) 13: 60.17% (462 questions) 14: 66.96% (345 questions) 15: 67.09% (237 questions) 16: 67.52% (117 questions) 17: 67.02% (94 questions) 18: 71.05% (76 questions) 19: 76.74% (43 questions) 20: 56.25% (32 questions) 21: 78.95% (19 questions) 22: 75.00% (12 questions) 23: 50.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

===================================== nowTime: 2019-10-22 10:06:53 Epoch: 11, Loss: 0.2924597249457694, Lr: 2.0000000000000003e-06 Elapsed time: 21957, Speed(s/batch): 1.3072959441433467

Binary: 77.43% Open: 40.87% Accuracy: 57.65% Distribution: 1.69 (lower is better) Accuracy / structural type: choose: 78.30% (1129 questions) compare: 67.23% (589 questions) logical: 75.98% (1803 questions) query: 40.87% (6805 questions) verify: 80.82% (2252 questions) Accuracy / semantic type: attr: 65.48% (5186 questions) cat: 45.17% (1149 questions) global: 57.32% (157 questions) obj: 87.15% (778 questions) rel: 48.38% (5308 questions) Accuracy / steps number: 1: 72.15% (237 questions) 2: 51.07% (6395 questions) 3: 61.28% (4266 questions) 4: 65.45% (793 questions) 5: 76.16% (822 questions) 6: 75.61% (41 questions) 7: 100.00% (20 questions) 8: 100.00% (3 questions) 9: 100.00% (1 questions) Accuracy / words number: 3: 34.44% (151 questions) 4: 50.63% (630 questions) 5: 43.64% (1290 questions) 6: 54.68% (2074 questions) 7: 56.58% (1642 questions) 8: 59.83% (1185 questions) 9: 62.84% (1281 questions) 10: 65.01% (1249 questions) 11: 59.96% (994 questions) 12: 63.17% (638 questions) 13: 59.31% (462 questions) 14: 66.67% (345 questions) 15: 66.67% (237 questions) 16: 67.52% (117 questions) 17: 63.83% (94 questions) 18: 68.42% (76 questions) 19: 79.07% (43 questions) 20: 53.12% (32 questions) 21: 68.42% (19 questions) 22: 66.67% (12 questions) 23: 25.00% (4 questions) 24: 100.00% (2 questions) 25: 100.00% (1 questions)

mrfarazi commented 4 years ago

Thanks for you reply @nbgao , did you use the same seed for mcan_small?

nbgao commented 4 years ago

Thanks for you reply @nbgao , did you use the same seed for mcan_small?

We use SEED=1016 for mcan_small in GQA.