search

clear-nus / edc

MIT License
32 stars 4 forks source link

Exp #5

Open MaskXman opened 3 weeks ago

MaskXman commented 3 weeks ago

Hello , I carefully your paper, and I try to your expermint,but the result just not fine. Could you please help me?,I use 500 instances in REBEL datasets, and I use Mistral-7b-v0.2 by all moduls. Total scores

Ent_type Correct: 511 Incorrect: 121 Partial: 0 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.07650482640527438 Recall: 0.08195619711299154 F1: 0.07827778659037991

Partial Correct: 507 Incorrect: 0 Partial: 125 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.08816058257322568 Recall: 0.09225983076157292 F1: 0.08940488193848074

Strict Correct: 418 Incorrect: 214 Partial: 0 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.06535711951341618 Recall: 0.06791936286709806 F1: 0.06617072988402008

Exact Correct: 507 Incorrect: 125 Partial: 0 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.08000430242839401 Recall: 0.08266135722581715 F1: 0.08084201070761549

Scores per tag

Subjects

Ent_type Correct: 236 Incorrect: 49 Partial: 0 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.11512361041977767 Recall: 0.1174713787954206 F1: 0.1157624025219844

Partial Correct: 245 Incorrect: 0 Partial: 40 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.12787041645926664 Recall: 0.13153310104529617 F1: 0.1290277086444334

Strict Correct: 209 Incorrect: 76 Partial: 0 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.10403185664509706 Recall: 0.10403185664509706 F1: 0.10403185664509706

Exact Correct: 245 Incorrect: 40 Partial: 0 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.11937945910071346 Recall: 0.12170233947237431 F1: 0.12015928322548532

Predicates

Ent_type Correct: 116 Incorrect: 3 Partial: 0 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04272045635908254 Recall: 0.04927824788451966 F1: 0.044269977022590265

Partial Correct: 96 Incorrect: 0 Partial: 23 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04207850387541776 Recall: 0.04716276754604281 F1: 0.04330286438300376

Strict Correct: 96 Incorrect: 23 Partial: 0 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04068991127229057 Recall: 0.04355400696864112 F1: 0.041340231584134024

Exact Correct: 96 Incorrect: 23 Partial: 0 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04068991127229057 Recall: 0.04355400696864112 F1: 0.041340231584134024

Objects

Ent_type Correct: 159 Incorrect: 69 Partial: 0 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.07582545213207234 Recall: 0.07715281234444997 F1: 0.0762734362037498

Partial Correct: 166 Incorrect: 0 Partial: 62 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.0894931143188983 Recall: 0.09619213539074166 F1: 0.09174547867927658

Strict Correct: 113 Incorrect: 115 Partial: 0 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.054878048780487805 Recall: 0.055251368840219015 F1: 0.05501908080305293

Exact Correct: 166 Incorrect: 62 Partial: 0 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.07570101211216194 Recall: 0.08138377302140368 F1: 0.07761738841878214

Full triple scores

Precision: 0.0052173913043478265 Recall: 0.0052173913043478265 F1: 0.0052173913043478265

bzhangj13zzz commented 3 weeks ago

Thanks for the update, I will try to address your issue with the OIE first, I will try to get back to you ASAP.

MaskXman commented 2 weeks ago

Hello，I try more times. Now I can arrive 40 But can not your fine result. This is my param and result. Could you please help me, Thank you

python run.py \ --oie_llm /home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2 \ --oie_few_shot_example_file_path ./few_shot_examples/webnlg/oie_few_shot_examples.txt \ --sd_llm /home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2 \ --sd_few_shot_example_file_path ./few_shot_examples/webnlg/sd_few_shot_examples.txt \ --sc_llm /home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2 \ --input_text_file_path ./datasets/webnlg.txt \ --output_dir /home/ubuntu/project/edc/output/Mistral-7B-Instruct-v0.2/webnlg \ --oie_refine_few_shot_example_file_path ./few_shot_examples/webnlg/oie_few_shot_refine_examples.txt \ --ee_llm /home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2 \ --target_schema ./schemas/webnlg_schema.csv \ --ee_few_shot_example_file_path ./few_shot_examples/webnlg/ee_few_shot_examples.txt

Total scores

Ent_type Correct: 6072 Incorrect: 261 Partial: 0 Missed: 5707 Spurious: 1822 Possible: 12040 Actual: 8155 Precision: 0.4748503528001334 Recall: 0.4913009913863156 F1: 0.4810109201639875

Partial Correct: 5677 Incorrect: 0 Partial: 656 Missed: 5707 Spurious: 1822 Possible: 12040 Actual: 8155 Precision: 0.473248927246977 Recall: 0.4859621322931903 F1: 0.4777925522182269

Strict Correct: 5447 Incorrect: 886 Partial: 0 Missed: 5707 Spurious: 1822 Possible: 12040 Actual: 8155 Precision: 0.4325468412621021 Recall: 0.44139850479440923 F1: 0.43586813302375293

Exact Correct: 5677 Incorrect: 656 Partial: 0 Missed: 5707 Spurious: 1822 Possible: 12040 Actual: 8155 Precision: 0.4511800283253233 Recall: 0.45991792621485456 F1: 0.45424457347479114

Scores per tag

Subjects

Ent_type Correct: 2186 Incorrect: 81 Partial: 0 Missed: 1746 Spurious: 374 Possible: 4013 Actual: 2641 Precision: 0.5202746627661303 Recall: 0.5297415894685519 F1: 0.5230781732488217

Partial Correct: 2113 Incorrect: 0 Partial: 154 Missed: 1746 Spurious: 374 Possible: 4013 Actual: 2641 Precision: 0.5262778319518934 Recall: 0.5315090199902487 F1: 0.527844141069397

Strict Correct: 2036 Incorrect: 231 Partial: 0 Missed: 1746 Spurious: 374 Possible: 4013 Actual: 2641 Precision: 0.4940069884609134 Recall: 0.4947586543149683 F1: 0.494254835039818

Exact Correct: 2113 Incorrect: 154 Partial: 0 Missed: 1746 Spurious: 374 Possible: 4013 Actual: 2641 Precision: 0.5126564277588168 Recall: 0.5135299853729888 F1: 0.5129449049244271

Predicates

Ent_type Correct: 1927 Incorrect: 7 Partial: 0 Missed: 2079 Spurious: 726 Possible: 4013 Actual: 2660 Precision: 0.46048675442873394 Recall: 0.4668454412481716 F1: 0.4622379327157484

Partial Correct: 1848 Incorrect: 0 Partial: 86 Missed: 2079 Spurious: 726 Possible: 4013 Actual: 2660 Precision: 0.45467658052982285 Recall: 0.4584349098000975 F1: 0.45576548025353486

Strict Correct: 1846 Incorrect: 88 Partial: 0 Missed: 2079 Spurious: 726 Possible: 4013 Actual: 2660 Precision: 0.44801316431009264 Recall: 0.4485616772306192 F1: 0.44822038030229155

Exact Correct: 1848 Incorrect: 86 Partial: 0 Missed: 2079 Spurious: 726 Possible: 4013 Actual: 2660 Precision: 0.4481756866569153 Recall: 0.44880546075085326 F1: 0.44841540711847877

Objects

Ent_type Correct: 1959 Incorrect: 173 Partial: 0 Missed: 1882 Spurious: 722 Possible: 4014 Actual: 2854 Precision: 0.45999106127092476 Recall: 0.4761092150170648 F1: 0.4644076060458313

Partial Correct: 1716 Incorrect: 0 Partial: 416 Missed: 1882 Spurious: 722 Possible: 4014 Actual: 2854 Precision: 0.44860840240533073 Recall: 0.4662969283276451 F1: 0.45405493255322604

Strict Correct: 1565 Incorrect: 567 Partial: 0 Missed: 1882 Spurious: 722 Possible: 4014 Actual: 2854 Precision: 0.38050544449861856 Recall: 0.38078985860555825 F1: 0.3806110840240533

Exact Correct: 1716 Incorrect: 416 Partial: 0 Missed: 1882 Spurious: 722 Possible: 4014 Actual: 2854 Precision: 0.40752478465789044 Recall: 0.416504144319844 F1: 0.4105395741914513

Full triple scores

Precision: 0.13698445539821 Recall: 0.06283098341211366 F1: 0.07798288587639099

bzhangj13zzz commented 2 weeks ago

Thanks for the information provided, I will try to reproduce the issues on my side and get back to you.

By the way, I noticed you are running experiments on WebNLG, are the issues with REBEL resolved?

MaskXman commented 2 weeks ago

I try to retuning in rebel many times, I can arrive SOTA in Rebel, even if can not arrive your result but I think It is my problem. Thank you!