search

ShannonAI / mrc-for-flat-nested-ner

Code for ACL 2020 paper `A Unified MRC Framework for Named Entity Recognition`
653 stars 117 forks source link

Bug in the flat-ner pre-processing #21

Open yhcc opened 4 years ago

yhcc commented 4 years ago

Hi, I found a bug in your pre-processing code. https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/a40c8b97cf081bbfcf845282012aad1012d7466f/data_preprocess/label_utils.py#L54 I think it should be

span_labels.append((start, len(sentence_tags) -1 , sentence_tags[-1].split("-")[-1]))

This bug will cause the number of entities decreases. In conll2003, each split has the following number of entities train: 23499 dev: 5942 test: 5648 Because of this bug, it will decrease to train: 22781 dev: 5796 test: 5465

I am not sure whether this will have an effect on the performance. I tried to run the corrected version by the configuration given in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , but in Telsa V100 (16G memory) it goes OOM. Therefore I have to use 'batch_size=16, gradient_accumulation_steps=2' to simulate, but the performance is far behind the reported one. Is there any trick to restrict the memory used or I need even bigger memory?

yhcc commented 4 years ago

BTW, the following are my outputs when I use ''batch_size=16, gradient_accumulation_steps=2'', I delete some of the middle outputs to make it shorter. In conll2003, usually (like LSTM+CRF) the F1 on the development set will be more than 3% higher than the test set (Besides, usually the F1 in the development set of BiLSTM+CRF version will achieve 95.0, but with 91.4 test set F1), and the following results are consistent with this. But in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , the gap is about 1.4% . Is there any possibility that you may use a corrected version of conll2003 like in those in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ?

{ "bert_frozen": "false", "hidden_size": 768, "hidden_dropout_prob": 0.2, "classifier_sign": "multi_nonlinear", "clip_grad": 1, "bert_config": { "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "type_vocab_size": 2, "vocab_size": 28996 }, "config_path": "/path/to/mrc_for_flat_nested_ner/config/en_bert_base_cased.json", "data_dir": "/home/ubuntu/exps/TransformerNER/data/conll2003/", "bert_model": "/path/to/bert-base-uncased/", "task_name": null, "max_seq_length": 150, "train_batch_size": 16, "dev_batch_size": 32, "test_batch_size": 32, "checkpoint": 600, "learning_rate": 3e-05, "num_train_epochs": 6, "warmup_proportion": -1.0, "local_rank": -1, "gradient_accumulation_steps": 2, "seed": 2333, "export_model": false, "output_dir": ".", "data_sign": "conll03", "weight_start": 1.0, "weight_end": 1.0, "weight_span": 1.0, "entity_sign": "flat", "n_gpu": 1, "dropout": 0.2, "entity_threshold": 0.5, "data_cache": false } -------------------- current data_sign: conll03 ==================== loading train data ... ... 56164 56164 train data loaded ==================== loading dev data ... ... 13000 13000 dev data loaded ==================== loading test data ... ... 13812 13812 test data loaded ###################################################################### EPOCH: 0 ------------------------------ current training loss is : 0.016730422154068947 ............................................................ DEV: loss, acc, precision, recall, f1 0.011 0.6385 0.4431 0.117 0.1851 SAVED model path is : ./bert_finetune_model_0_600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0104 0.652 0.4137 0.0996 0.1606 ------------------------------ ------------------------------ current training loss is : 0.002673722803592682 ............................................................ DEV: loss, acc, precision, recall, f1 0.0083 0.7126 0.6186 0.4184 0.4992 SAVED model path is : ./bert_finetune_model_0_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0093 0.7037 0.57 0.4091 0.4763 ------------------------------ ------------------------------ current training loss is : 0.0007827995577827096 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0049 0.8653 0.8349 0.7296 0.7787 SAVED model path is : ./bert_finetune_model_0_3000.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0053 0.8561 0.8078 0.7175 0.76 ------------------------------ ------------------------------ current training loss is : 0.006401577964425087 ............................................................ DEV: loss, acc, precision, recall, f1 0.0047 0.7522 0.6744 0.337 0.4494 ------------------------------ ------------------------------ current training loss is : 0.00014035690401215106 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.8934 0.8443 0.7814 0.8116 SAVED model path is : ./bert_finetune_model_0_6600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0038 0.8791 0.8159 0.7416 0.777 ------------------------------ ###################################################################### EPOCH: 1 current learning rate 2.8499999999999998e-05 current learning rate 2.8499999999999998e-05 ------------------------------ current training loss is : 0.018947187811136246 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8717 0.8382 0.7353 0.7834 ------------------------------ ------------------------------ current training loss is : [386/1723] 5.050747859058902e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0026 0.9438 0.9256 0.9055 0.9154 SAVED model path is : ./bert_finetune_model_1_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0039 0.9084 0.8754 0.8581 0.8667 ------------------------------ ------------------------------ current training loss is : 9.175582090392709e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9237 0.896 0.8175 0.8549 ------------------------------ ------------------------------ current training loss is : 0.0015588540118187666 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9103 0.8982 0.8307 0.8631 ------------------------------ ------------------------------ current training loss is : 0.00026114226784557104 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8382 0.8883 0.5875 0.7072 ------------------------------ ------------------------------ current training loss is : 0.00016665547445882112 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9406 0.9147 0.8547 0.8837 ------------------------------ ###################################################################### current learning rate 2.7074999999999997e-05 current learning rate 2.7074999999999997e-05 ------------------------------ current training loss is : 0.002201023744419217 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9519 0.9355 0.9003 0.9175 SAVED model path is : ./bert_finetune_model_2_2400.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0043 0.9279 0.9094 0.8656 0.8869 ------------------------------ ###################################################################### EPOCH: 3 current learning rate 2.5721249999999997e-05 current learning rate 2.5721249999999997e-05 ------------------------------ current training loss is : 4.75841952720657e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.9186 0.8595 0.8149 0.8366 ------------------------------ ------------------------------ current training loss is : 1.2278333088033833e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9512 0.9316 0.9067 0.9189 SAVED model path is : ./bert_finetune_model_3_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.004 0.9248 0.9031 0.8724 0.8875 ------------------------------ ------------------------------ current training loss is : 7.809271482983604e-05 current training loss is : 4.4565200369106606e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0057 0.9165 0.8801 0.8271 0.8528 ------------------------------ ------------------------------ current training loss is : 0.0001912666339194402 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9392 0.9026 0.8773 0.8898 ------------------------------ ------------------------------ current training loss is : 2.610034789540805e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0036 0.938 0.924 0.8722 0.8973 ###################################################################### EPOCH: 4 current learning rate 2.4435187499999995e-05 current learning rate 2.4435187499999995e-05 ------------------------------ current training loss is : 6.310163007583469e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9354 0.8781 0.8549 0.8663 ------------------------------ ------------------------------ current training loss is : 6.9439424805750605e-06 ............................................................ TEST: loss, acc, precision, recall, f1 0.0049 0.9332 0.916 0.8687 0.8917 ------------------------------ ------------------------------ current training loss is : 6.0556856624316424e-05 ------------------------------ ------------------------------ current training loss is : 4.879345578956418e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9415 0.9147 0.8654 0.8894 ------------------------------ ###################################################################### EPOCH: 5 current learning rate 2.3213428124999993e-05 current learning rate 2.3213428124999993e-05 ------------------------------ current training loss is : 0.00025477795861661434 ............................................................ DEV: loss, acc, precision, recall, f1 0.0038 0.9279 0.86 0.8468 0.8533 ------------------------------ ------------------------------ current training loss is : 9.94959282252239e-06 ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9525 0.9208 0.9046 0.9126 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9423 0.8931 0.8811 0.8871 ------------------------------ ------------------------------ current training loss is : 0.0022768585477024317 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9508 0.9247 0.892 0.9081 -----------------------------*- =&==&==&==&==&==&==&==&==&==&==&==&==&==&==&= Best DEV : overall best loss, acc, precision, recall, f1 0.0034 0.9566 0.939 0.9091 0.9238 scores on TEST when Best DEV:loss, acc, precision, recall, f1 0.0049 0.9332 0.916 0.8687 0.8917 =&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=

I add the following line in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/a40c8b97cf081bbfcf845282012aad1012d7466f/metric/flat_span_f1.py#L153 to make sure the loop will not stuck 

idx += 1
ghost commented 4 years ago

Thanks for asking! I will check the code and preprocessed datasets. And I will reply later. Thanks！

yhcc commented 4 years ago

Is there any progress for this bug ?

ghost commented 4 years ago

Sorry for the delay. I am dealing with an emergency. The final reply will be given by the 16th. Many many thanks! Hope you all the best.

yhcc commented 4 years ago

Thank you for your reply. Sorry for the pressure, the 16th is totally fine.

yhcc commented 4 years ago

Any update?

kamalkraj commented 4 years ago

BTW, the following are my outputs when I use ''batch_size=16, gradient_accumulation_steps=2'', I delete some of the middle outputs to make it shorter. In conll2003, usually (like LSTM+CRF) the F1 on the development set will be more than 3% higher than the test set (Besides, usually the F1 in the development set of BiLSTM+CRF version will achieve 95.0, but with 91.4 test set F1), and the following results are consistent with this. But in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , the gap is about 1.4% . Is there any possibility that you may use a corrected version of conll2003 like in those in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ?

{ "bert_frozen": "false", "hidden_size": 768, "hidden_dropout_prob": 0.2, "classifier_sign": "multi_nonlinear", "clip_grad": 1, "bert_config": { "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "type_vocab_size": 2, "vocab_size": 28996 }, "config_path": "/path/to/mrc_for_flat_nested_ner/config/en_bert_base_cased.json", "data_dir": "/home/ubuntu/exps/TransformerNER/data/conll2003/", "bert_model": "/path/to/bert-base-uncased/", "task_name": null, "max_seq_length": 150, "train_batch_size": 16, "dev_batch_size": 32, "test_batch_size": 32, "checkpoint": 600, "learning_rate": 3e-05, "num_train_epochs": 6, "warmup_proportion": -1.0, "local_rank": -1, "gradient_accumulation_steps": 2, "seed": 2333, "export_model": false, "output_dir": ".", "data_sign": "conll03", "weight_start": 1.0, "weight_end": 1.0, "weight_span": 1.0, "entity_sign": "flat", "n_gpu": 1, "dropout": 0.2, "entity_threshold": 0.5, "datacache": false } -------------------_- current datasign: conll03 ==================== loading train data ... ... 56164 56164 train data loaded ==================== loading dev data ... ... 13000 13000 dev data loaded ==================== loading test data ... ... 13812 13812 test data loaded ###################################################################### EPOCH: 0 ------------------------------ current training loss is : 0.016730422154068947 ............................................................ DEV: loss, acc, precision, recall, f1 0.011 0.6385 0.4431 0.117 0.1851 SAVED model path is : ./bert_finetune_model_0600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0104 0.652 0.4137 0.0996 0.1606 ------------------------------ -----------------------------_- current training loss is : 0.002673722803592682 ............................................................ DEV: loss, acc, precision, recall, f1 0.0083 0.7126 0.6186 0.4184 0.4992 SAVED model path is : ./bert_finetune_model_01200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0093 0.7037 0.57 0.4091 0.4763 ------------------------------ -----------------------------_- current training loss is : 0.0007827995577827096 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0049 0.8653 0.8349 0.7296 0.7787 SAVED model path is : ./bert_finetune_model_03000.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0053 0.8561 0.8078 0.7175 0.76 ------------------------------ ------------------------------ current training loss is : 0.006401577964425087 ............................................................ DEV: loss, acc, precision, recall, f1 0.0047 0.7522 0.6744 0.337 0.4494 ------------------------------ -----------------------------_- current training loss is : 0.00014035690401215106 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.8934 0.8443 0.7814 0.8116 SAVED model path is : ./bert_finetune_model_06600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0038 0.8791 0.8159 0.7416 0.777 ------------------------------ ###################################################################### EPOCH: 1 current learning rate 2.8499999999999998e-05 current learning rate 2.8499999999999998e-05 ------------------------------ current training loss is : 0.018947187811136246 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8717 0.8382 0.7353 0.7834 ------------------------------ -----------------------------_- current training loss is : [386/1723] 5.050747859058902e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0026 0.9438 0.9256 0.9055 0.9154 SAVED model path is : ./bert_finetune_model_11200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0039 0.9084 0.8754 0.8581 0.8667 ------------------------------ ------------------------------ current training loss is : 9.175582090392709e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9237 0.896 0.8175 0.8549 ------------------------------ ------------------------------ current training loss is : 0.0015588540118187666 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9103 0.8982 0.8307 0.8631 ------------------------------ ------------------------------ current training loss is : 0.00026114226784557104 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8382 0.8883 0.5875 0.7072 ------------------------------ ------------------------------ current training loss is : 0.00016665547445882112 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9406 0.9147 0.8547 0.8837 ------------------------------ ###################################################################### current learning rate 2.7074999999999997e-05 current learning rate 2.7074999999999997e-05 -----------------------------_- current training loss is : 0.002201023744419217 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9519 0.9355 0.9003 0.9175 SAVED model path is : ./bert_finetune_model_22400.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0043 0.9279 0.9094 0.8656 0.8869 ------------------------------ ###################################################################### EPOCH: 3 current learning rate 2.5721249999999997e-05 current learning rate 2.5721249999999997e-05 ------------------------------ current training loss is : 4.75841952720657e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.9186 0.8595 0.8149 0.8366 ------------------------------ -----------------------------_- current training loss is : 1.2278333088033833e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9512 0.9316 0.9067 0.9189 SAVED model path is : ./bert_finetune_model_31200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.004 0.9248 0.9031 0.8724 0.8875 ------------------------------ ------------------------------ current training loss is : 7.809271482983604e-05 current training loss is : 4.4565200369106606e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0057 0.9165 0.8801 0.8271 0.8528 ------------------------------ ------------------------------ current training loss is : 0.0001912666339194402 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9392 0.9026 0.8773 0.8898 ------------------------------ ------------------------------ current training loss is : 2.610034789540805e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0036 0.938 0.924 0.8722 0.8973 ###################################################################### EPOCH: 4 current learning rate 2.4435187499999995e-05 current learning rate 2.4435187499999995e-05 ------------------------------ current training loss is : 6.310163007583469e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9354 0.8781 0.8549 0.8663 ------------------------------ ------------------------------ current training loss is : 6.9439424805750605e-06 ............................................................ TEST: loss, acc, precision, recall, f1 0.0049 0.9332 0.916 0.8687 0.8917 ------------------------------ ------------------------------ current training loss is : 6.0556856624316424e-05 ------------------------------ ------------------------------ current training loss is : 4.879345578956418e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9415 0.9147 0.8654 0.8894 ------------------------------ ###################################################################### EPOCH: 5 current learning rate 2.3213428124999993e-05 current learning rate 2.3213428124999993e-05 ------------------------------ current training loss is : 0.00025477795861661434 ............................................................ DEV: loss, acc, precision, recall, f1 0.0038 0.9279 0.86 0.8468 0.8533 ------------------------------ ------------------------------ current training loss is : 9.94959282252239e-06 ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9525 0.9208 0.9046 0.9126 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9423 0.8931 0.8811 0.8871 ------------------------------ ------------------------------ current training loss is : 0.0022768585477024317 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9508 0.9247 0.892 0.9081 -----------------------------*- =&==&==&==&==&==&==&==&==&==&==&==&==&==&==&= Best DEV : overall best loss, acc, precision, recall, f1 0.0034 0.9566 0.939 0.9091 0.9238 scores on TEST when Best DEV:loss, acc, precision, recall, f1 0.0049 0.9332 0.916 0.8687 0.8917 =&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=

I add the following line in

https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/a40c8b97cf081bbfcf845282012aad1012d7466f/metric/flat_span_f1.py#L153

to make sure the loop will not stuck

idx += 1

@yhcc What you mean by the corrected version of the ConLL2003 dataset?

yhcc commented 4 years ago

@kamalkraj Since the original conll2003 data has some wrong annotation, some papers (as some listed in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ) have tried to revise these annotation errors, and released the revised dataset. If the revised data is used, usually the performance will be higher than previous work.

kamalkraj commented 4 years ago

@yhcc Thanks. Are you able to reproduce the results for the ConLL2003 dataset?

yhcc commented 4 years ago

No, I tried to use the code in this repo(revised the bug as post here), and get the results as I have post before. I also tried to rewrite this code from scratch, and got similar results as I post. And my log (based on their code) is quite different from their optimization log in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , and I am not sure whether I did something wrong or the bug caused the difference.

ghost commented 4 years ago

We will get back in one day or two. Sincerely sorry for the delay. We were busy with the corefQA release, the other repo.

yhcc commented 4 years ago

Looking forward to your reply

ghost commented 4 years ago

BTW, the following are my outputs when I use ''batch_size=16, gradient_accumulation_steps=2'', I delete some of the middle outputs to make it shorter. In conll2003, usually (like LSTM+CRF) the F1 on the development set will be more than 3% higher than the test set (Besides, usually the F1 in the development set of BiLSTM+CRF version will achieve 95.0, but with 91.4 test set F1), and the following results are consistent with this. But in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , the gap is about 1.4% . Is there any possibility that you may use a corrected version of conll2003 like in those in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ?

{ "bert_frozen": "false", "hidden_size": 768, "hidden_dropout_prob": 0.2, "classifier_sign": "multi_nonlinear", "clip_grad": 1, "bert_config": { "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "type_vocab_size": 2, "vocab_size": 28996 }, "config_path": "/path/to/mrc_for_flat_nested_ner/config/en_bert_base_cased.json", "data_dir": "/home/ubuntu/exps/TransformerNER/data/conll2003/", "bert_model": "/path/to/bert-base-uncased/", "task_name": null, "max_seq_length": 150, "train_batch_size": 16, "dev_batch_size": 32, "test_batch_size": 32, "checkpoint": 600, "learning_rate": 3e-05, "num_train_epochs": 6, "warmup_proportion": -1.0, "local_rank": -1, "gradient_accumulation_steps": 2, "seed": 2333, "export_model": false, "output_dir": ".", "data_sign": "conll03", "weight_start": 1.0, "weight_end": 1.0, "weight_span": 1.0, "entity_sign": "flat", "n_gpu": 1, "dropout": 0.2, "entity_threshold": 0.5, "datacache": false } -------------------_- current datasign: conll03 ==================== loading train data ... ... 56164 56164 train data loaded ==================== loading dev data ... ... 13000 13000 dev data loaded ==================== loading test data ... ... 13812 13812 test data loaded ###################################################################### EPOCH: 0 ------------------------------ current training loss is : 0.016730422154068947 ............................................................ DEV: loss, acc, precision, recall, f1 0.011 0.6385 0.4431 0.117 0.1851 SAVED model path is : ./bert_finetune_model_0600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0104 0.652 0.4137 0.0996 0.1606 ------------------------------ -----------------------------_- current training loss is : 0.002673722803592682 ............................................................ DEV: loss, acc, precision, recall, f1 0.0083 0.7126 0.6186 0.4184 0.4992 SAVED model path is : ./bert_finetune_model_01200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0093 0.7037 0.57 0.4091 0.4763 ------------------------------ -----------------------------_- current training loss is : 0.0007827995577827096 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0049 0.8653 0.8349 0.7296 0.7787 SAVED model path is : ./bert_finetune_model_03000.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0053 0.8561 0.8078 0.7175 0.76 ------------------------------ ------------------------------ current training loss is : 0.006401577964425087 ............................................................ DEV: loss, acc, precision, recall, f1 0.0047 0.7522 0.6744 0.337 0.4494 ------------------------------ -----------------------------_- current training loss is : 0.00014035690401215106 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.8934 0.8443 0.7814 0.8116 SAVED model path is : ./bert_finetune_model_06600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0038 0.8791 0.8159 0.7416 0.777 ------------------------------ ###################################################################### EPOCH: 1 current learning rate 2.8499999999999998e-05 current learning rate 2.8499999999999998e-05 ------------------------------ current training loss is : 0.018947187811136246 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8717 0.8382 0.7353 0.7834 ------------------------------ -----------------------------_- current training loss is : [386/1723] 5.050747859058902e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0026 0.9438 0.9256 0.9055 0.9154 SAVED model path is : ./bert_finetune_model_11200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0039 0.9084 0.8754 0.8581 0.8667 ------------------------------ ------------------------------ current training loss is : 9.175582090392709e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9237 0.896 0.8175 0.8549 ------------------------------ ------------------------------ current training loss is : 0.0015588540118187666 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9103 0.8982 0.8307 0.8631 ------------------------------ ------------------------------ current training loss is : 0.00026114226784557104 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8382 0.8883 0.5875 0.7072 ------------------------------ ------------------------------ current training loss is : 0.00016665547445882112 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9406 0.9147 0.8547 0.8837 ------------------------------ ###################################################################### current learning rate 2.7074999999999997e-05 current learning rate 2.7074999999999997e-05 -----------------------------_- current training loss is : 0.002201023744419217 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9519 0.9355 0.9003 0.9175 SAVED model path is : ./bert_finetune_model_22400.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0043 0.9279 0.9094 0.8656 0.8869 ------------------------------ ###################################################################### EPOCH: 3 current learning rate 2.5721249999999997e-05 current learning rate 2.5721249999999997e-05 ------------------------------ current training loss is : 4.75841952720657e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.9186 0.8595 0.8149 0.8366 ------------------------------ -----------------------------_- current training loss is : 1.2278333088033833e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9512 0.9316 0.9067 0.9189 SAVED model path is : ./bert_finetune_model_31200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.004 0.9248 0.9031 0.8724 0.8875 ------------------------------ ------------------------------ current training loss is : 7.809271482983604e-05 current training loss is : 4.4565200369106606e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0057 0.9165 0.8801 0.8271 0.8528 ------------------------------ ------------------------------ current training loss is : 0.0001912666339194402 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9392 0.9026 0.8773 0.8898 ------------------------------ ------------------------------ current training loss is : 2.610034789540805e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0036 0.938 0.924 0.8722 0.8973 ###################################################################### EPOCH: 4 current learning rate 2.4435187499999995e-05 current learning rate 2.4435187499999995e-05 ------------------------------ current training loss is : 6.310163007583469e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9354 0.8781 0.8549 0.8663 ------------------------------ ------------------------------ current training loss is : 6.9439424805750605e-06 ............................................................ TEST: loss, acc, precision, recall, f1 0.0049 0.9332 0.916 0.8687 0.8917 ------------------------------ ------------------------------ current training loss is : 6.0556856624316424e-05 ------------------------------ ------------------------------ current training loss is : 4.879345578956418e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9415 0.9147 0.8654 0.8894 ------------------------------ ###################################################################### EPOCH: 5 current learning rate 2.3213428124999993e-05 current learning rate 2.3213428124999993e-05 ------------------------------ current training loss is : 0.00025477795861661434 ............................................................ DEV: loss, acc, precision, recall, f1 0.0038 0.9279 0.86 0.8468 0.8533 ------------------------------ ------------------------------ current training loss is : 9.94959282252239e-06 ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9525 0.9208 0.9046 0.9126 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9423 0.8931 0.8811 0.8871 ------------------------------ ------------------------------ current training loss is : 0.0022768585477024317 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9508 0.9247 0.892 0.9081 -----------------------------*- =&==&==&==&==&==&==&==&==&==&==&==&==&==&==&= Best DEV : overall best loss, acc, precision, recall, f1 0.0034 0.9566 0.939 0.9091 0.9238 scores on TEST when Best DEV:loss, acc, precision, recall, f1 0.0049 0.9332 0.916 0.8687 0.8917 =&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=

I add the following line in

https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/a40c8b97cf081bbfcf845282012aad1012d7466f/metric/flat_span_f1.py#L153

to make sure the loop will not stuck

idx += 1

Thanks for pointing out my mistake. Yes, this is a bug introduced when I was trying to clean my codebase. I fixed it in the commit (f80ed26). Please pull the latest repo.

ghost commented 4 years ago

Hi, I found a bug in your pre-processing code. https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/a40c8b97cf081bbfcf845282012aad1012d7466f/data_preprocess/label_utils.py#L54

I think it should be

span_labels.append((start, len(sentence_tags) -1 , sentence_tags[-1].split("-")[-1]))

This bug will cause the number of entities decreases. In conll2003, each split has the following number of entities train: 23499 dev: 5942 test: 5648 Because of this bug, it will decrease to train: 22781 dev: 5796 test: 5465

I am not sure whether this will have an effect on the performance. I tried to run the corrected version by the configuration given in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , but in Telsa V100 (16G memory) it goes OOM. Therefore I have to use 'batch_size=16, gradient_accumulation_steps=2' to simulate, but the performance is far behind the reported one. Is there any trick to restrict the memory used or I need even bigger memory?

Thanks for pointing out my mistake. Yes, this is a bug introduced when I was trying to clean my codebase. I fixed it in the commit (f80ed26).

ghost commented 4 years ago

No, I tried to use the code in this repo(revised the bug as post here), and get the results as I have post before. I also tried to rewrite this code from scratch, and got similar results as I post. And my log (based on their code) is quite different from their optimization log in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , and I am not sure whether I did something wrong or the bug caused the difference.

We used the official released CoNLL-2003 train/dev/test sets for experiments.

ghost commented 4 years ago

No, I tried to use the code in this repo(revised the bug as post here), and get the results as I have post before. I also tried to rewrite this code from scratch, and got similar results as I post. And my log (based on their code) is quite different from their optimization log in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , and I am not sure whether I did something wrong or the bug caused the difference.

Since we update the codebase, I run the experiments with the code (f80ed26) and will update the log for CoNLL-2003 tomorrow. Many thanks.

ghost commented 4 years ago

I replicated the experiment result on English CoNLL03 with the latest code (f80ed26). And configurations can be found in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt.

Please contact me if you have any further questions. Many thanks and sorry for the late reply.

yhcc commented 4 years ago

Thanks for your reply. BERT-large can achieve 93+ makes sense.

yhcc commented 4 years ago

Did the results in the English Ontonotes also use the BERT-large model ?

ghost commented 4 years ago

Did the results in the English Ontonotes also use the BERT-large model ?

Sorry for the late reply. Yes. The result of the English Ontonotes is obtained by finetuning the BERT-Large model.

EmanuelaBoros commented 3 years ago

I replicated the experiment result on English CoNLL03 with the latest code (f80ed26). And configurations can be found in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt.

Please contact me if you have any further questions. Many thanks and sorry for the late reply.

I was not able to reproduce the results reported in the ACL paper for the CoNLL 2003. Would it be possible to share the reproduce script for this dataset also? Thanks.