juntaoy
commented
4 years ago @zhaoxf4 I've checked your data, they are identical to mine and I also tested on my best model they give the same results I reported in the paper. The log is attached below. I got at least 93 over 5 runs, 92.1 is way too low:) I can find three differences:
-
the fasttext embedding, but I don't think this would be a large issue if you retrain the system yourself. The cc.en.300.vec.filtered is just a subset of the cc.en.300.vec that only contains words appeared in the dataset, so that it loads faster. The reason you got the error is you need to remove the first line of the cc.en.300.vec which is not an embedding.
-
I train the system using the concatenation of train and dev set and without early stopping.
-
Due to typos in the early version of the extract_bert_features.sh, the --window_size I actually used is not 128 but its default value 511 in extract_features.py , so you need change it to 511 in order to get the same results. I've just updated the extract_bert_features.sh to make the --window_size = 511 to avoid confusing.
WARNING:tensorflow:From` /homes/juntao/miniconda2/envs/tf15/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
Running experiment: eng_conll03
ffnn_size = 150
ffnn_depth = 2
filter_widths = [
3
4
5
]
filter_size = 50
char_embedding_size = 8
char_vocab_path = "char_vocab.eng.conll03.txt"
context_embeddings {
path = "../fasttext/cc.en.300.vec.filtered"
size = 300
}
contextualization_size = 200
contextualization_layers = 3
lm_size = 1024
lm_layers = 4
lm_path = "../cogsci/bert-model/bert_conll03_features.hdf5"
max_gradient_norm = 5.0
lstm_dropout_rate = 0.4
lexical_dropout_rate = 0.5
dropout_rate = 0.2
optimizer = "adam"
learning_rate = 0.001
decay_rate = 0.999
decay_frequency = 100
train_path = "train_dev.conll03.jsonlines"
eval_path = ""
test_path = "test.yu.json"
ner_types = [
"ORG"
"MISC"
"PER"
"LOC"
]
eval_frequency = 500
report_frequency = 100
log_root = "logs"
max_step = 80000
flat_ner = true
log_dir = "logs/eng_conll03"
Loading word embeddings from ../fasttext/cc.en.300.vec.filtered...
Done loading word embeddings.
use_lee_lstm
/homes/juntao/miniconda2/envs/tf15/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2020-10-26 11:15:59.377855: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-26 11:15:59.683691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:09:00.0
totalMemory: 10.92GiB freeMemory: 10.77GiB
2020-10-26 11:15:59.683785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2020-10-26 11:16:00.103129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-26 11:16:00.103192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2020-10-26 11:16:00.103212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2020-10-26 11:16:00.103399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10428 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:09:00.0, compute capability: 6.1)
Restoring from logs/eng_conll03/model.max.ckpt
Loaded 231 eval examples.
Evaluated 1/231 examples.
Evaluated 11/231 examples.
Evaluated 21/231 examples.
Evaluated 31/231 examples.
Evaluated 41/231 examples.
Evaluated 51/231 examples.
Evaluated 61/231 examples.
Evaluated 71/231 examples.
Evaluated 81/231 examples.
Evaluated 91/231 examples.
Evaluated 101/231 examples.
Evaluated 111/231 examples.
Evaluated 121/231 examples.
Evaluated 131/231 examples.
Evaluated 141/231 examples.
Evaluated 151/231 examples.
Evaluated 161/231 examples.
Evaluated 171/231 examples.
Evaluated 181/231 examples.
Evaluated 191/231 examples.
Evaluated 201/231 examples.
Evaluated 211/231 examples.
Evaluated 221/231 examples.
Evaluated 231/231 examples.
Time used: 29 second, 1588.77 w/s
test Mention F1: 93.46%
test Mention recall: 93.31%
test Mention precision: 93.62%
93.6&93.3&93.5
****************SUB NER TYPES********************
ORG F1: 92.51%
ORG recall: 93.02%
ORG precision: 92.02%
MISC F1: 84.77%
MISC recall: 84.47%
MISC precision: 85.08%
PER F1: 97.55%
PER recall: 97.40%
PER precision: 97.70%
LOC F1: 94.11%
LOC recall: 93.35%
LOC precision: 94.88%
wangxinyu0922
zhaoxf4
lzf00
I'm sorry to bother you but I couldn't replicate same scores on en_conll2003 dataset. I only reproduced to 92.12, 1.4 lower than yours. I check my dataset and ensure the labels are the same as conll2003 annotate software. So I think my en_conll2003 is complete. This dataset is download from here . Then I write a script named "data_process.py"(python3) to convert conllx format to your dict format. I use '-DOCSTART- -X- -X- O' to split it. I think this script is no problem because there is no error in extract_feature.sh and evaluate.py and I also check the processed dataset. so I think the difference between your dataset and mine is very small.
environment:
the only difference is I replace fastText with glove 6B because I couldn't find fasttext/cc.en.300.vec.filtered, I just find a vector file named "cc.en.300.vec" from fastText. I met a error when I try "cc.en.300.vec", this is the first I try fastText and I don't know what happened. The error log ls as follow:
Then I continue to use glove 6B because I think embedding shouldn't have huge impact. This is the log of the en_conll2003 Pre-trained models from your link "acl2020 best models":
This is the log of the model I train again:
The raw_en_conll2003 and the en_conll2003 I processed and the data_process.py(python3) are in sharelink1 or sharelink2, and the extract_feature.sh and experiments.conf I used also in it. You can check it.
Do you know what's wrong with me? The dataset, the fastText embedding, or the hyperparameter? Can you help me? Thank you.