search

raspberryice / gen-arg

Code for paper "Document-Level Argument Extraction by Conditional Generation". NAACL 21'
MIT License
115 stars 29 forks source link

lose RAMS dataset file? #29

Open caddy-liu opened 1 year ago

caddy-liu commented 1 year ago

Hi.

I tried to follow the code on dataset RAMS.And I changed the argument called 'test-file' in _convert_gen_tooutput.py to 'data/RAMS_1.0/data/test.jsonlines' when evaluating head score but only received low performance as follow:

Precision: 33.5640 Recall: 36.2133 F1: 34.8384.

I did't find the file called 'test_head_coref.jsonlines' and 'test_head.jsonlines'. Are they include in RAMS or created by yourself?

It puzzled me. Did I downloaded wrong version? My version: RAMS_1.0c.tar.gz [current as of March 2023, 4.5MB].

Thanks for reply.

raspberryice commented 1 year ago

The two files are processed by us. See the end of the README.md for instructions on how to download these files. Note that only Span F1 (should be the original exact match score) and Head F1 (match of head words) are reported in our paper.

We reported these metrics to match a previous paper "A two-step approach for implicit event argument detection.", not the original RAMS paper because the evaluation setting was different.

caddy-liu commented 1 year ago

Thank you very much for your answer！