Open YanLiang1102 opened 6 years ago
YanLiang1102
commented
6 years ago 99 0.000 3.572 0.000 57.933 49.355 53.302 91.894 85.899 1438.3 0.0 iteration result using eval_data 100 times ,the previosu result is iteration 30 times, so more iteration not helping
YanLiang1102
commented
6 years ago YanLiang1102
commented
6 years ago After add in the ANERCorp here is the accuracy:
29 0.000 11.407 0.000 58.158 50.018 53.782 91.894 85.899 6835.0 0.0
this is because out of 150k entity tokens, 88% of them are useless 'O' object so our performance not get enhanced @ahalterman @cegme
tags accuracy goes down a little bit.
so LDC+ANERCORP with no merged class, but fasttext pretrained embedding 29 0.000 10.027 0.000 56.598 51.344 53.843 91.894 85.899 13987.00.0
when trained by 10 times got a better performance: 9 0.000 270.721 0.000 56.944 52.855 54.823 91.894 85.899 13959.30.0
ahalterman
commented
6 years ago Well, not exactly "useless"..., since we need to be able to distinguish between entities and non-entities.
What are the different numbers in accuracy? Does each one represent a tag type? We need to figure out how to handle ANER not having a full range of labels that OntoNotes has. One way would be to go from spaCy format to Prodigy format, where each task is one single entity label, rather than all highlighted entities. Then when we use the more limited ANER data, we're not incorrectly telling it there's no entity there when there actually is.
It would also be a cool experiment to know whether "Prodigy-style" training underperforms spaCy training (and by how much).
YanLiang1102
commented
6 years ago @ahalterman @cegme so the token accuracy from 58.589 to 58.158 but token accuracy got enhanced from 49.871 to 50.018. for the difference of these two take a look at here. http://web.stanford.edu/class/cs224n/assignment3/assignment3.pdf
YanLiang1102
commented
6 years ago http://users.dsic.upv.es/~ybenajiba/downloads.html I am thinking using the ANERGazet using thsi to filter out the entity in some arabic docs we have and train the model based on that. so the ANER data is transferred from ANER format to BILOU format , and yes each word has a tag on it, what does the prodigy one looks like? and all the labeled tags has been transfered into one large documents, here is what the data looks like. https://raw.githubusercontent.com/oudalab/Arabic-NER/master/data/ANERCorp_conll_new.ner.json
some related stuff I found similar to what you are talking about @ahalterman https://support.prodi.gy/t/remarkable-difference-between-prodigy-and-custom-training-times/467/3
YanLiang1102
commented
6 years ago pretrained embedding stuff? @ahalterman wonder if this is what u are talking about. or u have better examples? https://github.com/explosion/spaCy/issues/2084
YanLiang1102
commented
6 years ago YanLiang1102
commented
6 years ago to avoid the catastrophic problem, our plan is to using spaCy and ldc+anercorp and we get the model, then we change the ldc+aner data into prodigy style and we update the model using the (ldc+anercorp+prodigy user labeled data) need to find the model how to update the model (probably in prodigy) or some cli code to do that @ahalterman
YanLiang1102
commented
6 years ago Performance with pretrained embedding and merged tagged class is this: as the best:
'dep_loss' 'tag_loss' 'uas' 'tags_acc' 'token_acc' 'ents_p' 'ents_r' 'ents_f' 'cpu_wps' 'gpu_wps'
0.000 275.748 0.000 58.406 54.254 56.254 91.894 85.899 15447.2 0.0token accuracy is 58.406 and entity accuracy is 54.254 and our base line is : 58.158 50.018 (with anercorp+ldc, no pretrained embedding, no merge ner tags) with anercorp+ldc, with merge tage with pretrained embedding token acurracy is similar from 58.406 drop to 58.158 and the entity accuracy enhanced a lot from 50.018 to 54.254. @ahalterman
ahalterman
commented
6 years ago Can you add the header to indicate what the 11 numbers mean?
YanLiang1102
commented
6 years ago yeah it is at the top of this issue: and also here 'itr','dep_loss', 'tag_loss', 'uas', 'tags_acc', 'token_acc', 'ents_p', 'ents_r', 'ents_f', 'cpu_wps', 'gpu_wps' eval_data=eval_data
YanLiang1102
commented
6 years ago training data tag distribution: {'B-LOC': 1889, 'B-MISC': 3521, 'B-ORG': 3912, 'B-PERSON': 6014, 'I-LOC': 744, 'I-MISC': 3812, 'I-ORG': 3630, 'I-PERSON': 1665, 'L-LOC': 1875, 'L-MISC': 3518, 'L-ORG': 3891, 'L-PERSON': 5982, 'O': 358866, 'U-LOC': 7323, 'U-MISC': 7264, 'U-ORG': 2576, 'U-PERSON': 3149} and test data tag distribution: {'B-LOC': 168, 'B-MISC': 307, 'B-ORG': 341, 'B-PERSON': 383, 'I-LOC': 46, 'I-MISC': 327, 'I-ORG': 465, 'I-PERSON': 93, 'L-LOC': 169, 'L-MISC': 307, 'L-ORG': 337, 'L-PERSON': 376, 'O': 27441, 'U-LOC': 384, 'U-MISC': 765, 'U-ORG': 173, 'U-PERSON': 204}
https://github.com/izarov/cs224n/blob/master/assignment3/handouts/assignment3-soln.pdf the confusion matrix for NER analysis is a good way to go to check which tags are really get misunderstood!
spaCy training output
exception that during training spaCy throw, and I made the code to eat the exception