stevenlujpl
commented
2 years ago @wkiri I've updated the parser scripts to auto-annotate words using a gazette file. The changes I made are currently in target-gazette branch. I tested the changes using one document, and everything looks correct to me. I will test with more docs next week, and merge the changes to the master branch.
I added a command line argument -g or --gazette_file to the corenlp_parser.py, jsre_parser.py, paper_parser.py, jgr_parser.py, and lpsc_parser.py. The -g or --gazette_file argument is optional. If a gazette file is provided, we will do gazette target matching and add the matching targets to the jsonl file. Otherwise, the gazette target marching will be disabled. The gazette file must consist of "entity_type entity_name". It is ok to have multiple entity types (e.g., Element, Mineral, Target), and only Target entities will be used.
The format of the gazette targets written to the jsonl file is similar to the format of NER targets. The only difference is the source field. The source fields of gazette targets will have the value gazette, and the source fields of CoreNLP targets will have the value corenlp. Please see the following gazette targets as examples:
{'text': u'Rocknest', 'begin': 1932, 'end': 1940, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 1987, 'end': 1995, 'source': 'gazette', 'label': 'Target'}
{'text': u'RMI', 'begin': 2841, 'end': 2844, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 3366, 'end': 3374, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 4426, 'end': 4434, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 4474, 'end': 4482, 'source': 'gazette', 'label': 'Target'}
{'text': u'RMI', 'begin': 4629, 'end': 4632, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 5290, 'end': 5298, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 5376, 'end': 5384, 'source': 'gazette', 'label': 'Target'}
{'text': u'Confidence Hills', 'begin': 6715, 'end': 6731, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 6801, 'end': 6809, 'source': 'gazette', 'label': 'Target'}
{'text': u'Rocknest', 'begin': 6903, 'end': 6911, 'source': 'gazette', 'label': 'Target'}The test I did was using one MSL 2015 doc (2015_2767.pdf) and the gazette file at/home/youlu/MTE/MTE/ref/targets_minerals-2017-05_elements.gaz.txt. The result jsonl file is at /home/youlu/MTE/working_dir/target_gazette/2767_gazette.jsonl. The gazette targets listed above are found in my test run. These gazette targets will be filtered by the NER targets. If a target is already detected by CoreNLP, it won't be written to the jsonl file. In this case, Confidence Hills isn't written to the jsonl file because CoreNLP has already detected it as a Target.
wkiri
We want to increase recall of Target types specifically. One way is to do a string-based matching from a gazette of entity terms. We can provide a gazette file that consists of "Entity_type Entity_name" pairs to inform the string matching.
Steven suggests integrating this capability into
corenlp_parser.py. It would happen after applying the trained NER model and before running relation extraction.-g <gazette_file>which if specified would activate this capabilitypre_annotate.pyscript can be an inspiration for this capability (but does not need to be followed exactly).