High accuracy for CoLA task

[pfecht]

I try to reproduce the CoLA results from the BERT paper (BERTBase, Single GPU).

Running the following command

python run_classifier.py \
  --task_name cola \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/CoLA/ \
  --bert_model bert-base-uncased \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir $OUT_DIR/cola_output/

I get eval results of

12/16/2018 12:31:34 - INFO - __main__ -   ***** Eval results *****
12/16/2018 12:31:34 - INFO - __main__ -     eval_accuracy = 0.8302972195589645
12/16/2018 12:31:34 - INFO - __main__ -     eval_loss = 0.5117322660925734
12/16/2018 12:31:34 - INFO - __main__ -     global_step = 804
12/16/2018 12:31:34 - INFO - __main__ -     loss = 0.17348005173644468

An accuracy of 0.83 would be fantastic, but compared to the 0.521 stated in the paper this doesn't seem very realistic.

Any suggestions what I'm doing wrong?

[davidefiocco]

The metric used for evaluation of CoLA in the GLUE benchmark is not accuracy but the https://en.wikipedia.org/wiki/Matthews_correlation_coefficient (see https://gluebenchmark.com/tasks). Indeed authors report in https://arxiv.org/abs/1810.04805 0.521 for Matthews correlation with BERT-base.

[pfecht]

Makes sense, looks like I missed that point. Thank you.