search

clear-nus / edc

MIT License
32 stars 4 forks source link

OIE Question #4

Open MaskXman opened 1 month ago

MaskXman commented 1 month ago

Sorry， I run example dataset，and I find I can not oie_triplets so that output.txt is all the[ ] image image Could you please tell me why?,and how can I deal . Thank you very much!!!!!

bzhangj13zzz commented 1 month ago

Can you let me know which models you are using?

MaskXman commented 1 month ago

Of course, I used image

bzhangj13zzz commented 1 month ago

Thanks, I haven't tried this model, I will get back to you later

MaskXman commented 1 month ago

Thank you!!!!!!!!

bzhangj13zzz commented 1 month ago

Hi, I have tried the mistral7b-v0.3 model, the OIE stage seems to be working just fine on my side. Can you please share the detailed parameters and data you are using? From the error messages in your screenshots, it seems to be a parsing issue - the function that extracts a list of triplets from the LLM's raw output.

I will also add better logging to this repo for better debugging soon.

MaskXman commented 1 month ago

Thanks，I tried to use the v0.2 models and this working just fine, But I used llama3、vicuna-7b-v1.5 models. I find this working just not fine, I can not find this reason. These models are orgin from huggingface, I dont fine-tuning. I just test this project. Finally, the datatest is rebel which this project provide.

bzhangj13zzz commented 1 month ago

Ok, I will look into it

MaskXman commented 1 month ago

Thanks , If you want to ask me to provide anything, I will do , It is my pleasure.

bzhangj13zzz commented 1 month ago

That is really odd, I have tired the models you mentioned and they all work fine, at least in the OIE stage. Can you please share the exact command you run? For example, something like:

python run.py \ --oie_llm lmsys/vicuna-7b-v1.5 \ --oie_few_shot_example_file_path ./few_shot_examples/webnlg/oie_few_shot_examples.txt \ --sd_llm lmsys/vicuna-7b-v1.5 \ --sd_few_shot_example_file_path ./few_shot_examples/webnlg/sd_few_shot_examples.txt \ --sc_llm lmsys/vicuna-7b-v1.5 \ --input_text_file_path ./datasets/example.txt \ --output_dir ./output/example \ --oie_refine_few_shot_example_file_path ./few_shot_examples/webnlg/oie_few_shot_refine_examples.txt \ --ee_llm lmsys/vicuna-7b-v1.5 \ --enrich_schema \ --ee_few_shot_example_file_path ./few_shot_examples/webnlg/ee_few_shot_examples.txt

MaskXman commented 1 month ago

of course, But I directly run not Terminal, such as:

from argparse import ArgumentParser from edc.edc_framework import EDC import os

os.environ["CUDA_VISIBLE_DEVICES"] = '1,2,3,5,6,7' os.environ["TOKENIZERS_PARALLELISM"] = "false"

if name == "main": parser = ArgumentParser()

OIE module setting

parser.add_argument(
    "--oie_llm", default="/home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2", help="LLM used for open information extraction."
)
parser.add_argument(
    "--oie_prompt_template_file_path",
    default="/home/ubuntu/project/edc/prompt_templates/oie_template.txt",
    help="Promp template used for open information extraction.",
)
parser.add_argument(
    "--oie_few_shot_example_file_path",
    default="/home/ubuntu/project/edc/few_shot_examples/default/ee_few_shot_examples.txt",
    help="Few shot examples used for open information extraction.",
)

# Schema Definition setting
parser.add_argument(
    "--sd_llm", default="/home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2", help="LLM used for schema definition."
)
parser.add_argument(
    "--sd_prompt_template_file_path",
    default="/home/ubuntu/project/edc/prompt_templates/sd_template.txt",
    help="Prompt template used for schema definition.",
)
parser.add_argument(
    "--sd_few_shot_example_file_path",
    default="/home/ubuntu/project/edc/few_shot_examples/default/sd_few_shot_examples.txt",
    help="Few shot examples used for schema definition.",
)

# Schema Canonicalization setting
parser.add_argument(
    "--sc_llm",
    default="/home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2",
    help="LLM used for schema canonicaliztion verification.",
)
parser.add_argument(
    "--sc_prompt_template_file_path",
    default="/home/ubuntu/project/edc/prompt_templates/sc_template.txt",
    help="Prompt template used for schema canonicalization verification.",
)

# Refinement setting
parser.add_argument("--sr_adapter_path", default=None, help="Path to adapter of schema retriever.")
parser.add_argument(
    "--oie_refine_prompt_template_file_path",
    default="/home/ubuntu/project/edc/prompt_templates/oie_r_template.txt",
    help="Prompt template used for refined open information extraction.",
)
parser.add_argument(
    "--oie_refine_few_shot_example_file_path",
    default="/home/ubuntu/project/edc/few_shot_examples/default/oie_few_shot_refine_examples.txt",
    help="Few shot examples used for refined open information extraction.",
)
parser.add_argument(
    "--ee_llm", default="/home/ubuntu/llm_models/Mistral-7B-Instruct-v0.2", help="LLM used for entity extraction."
)
parser.add_argument(
    "--ee_prompt_template_file_path",
    default="/home/ubuntu/project/edc/prompt_templates/ee_template.txt",
    help="Prompt templated used for entity extraction.",
)
parser.add_argument(
    "--ee_few_shot_example_file_path",
    default="/home/ubuntu/project/edc/few_shot_examples/default/ee_few_shot_examples.txt",
    help="Few shot examples used for entity extraction.",
)
parser.add_argument(
    "--em_prompt_template_file_path",
    default="/home/ubuntu/project/edc/prompt_templates/em_template.txt",
    help="Prompt template used for entity merging.",
)

# Input setting
parser.add_argument(
    "--input_text_file_path",
    default="/home/ubuntu/project/edc/datasets/rebel.txt",
    help="File containing input texts to extract KG from, each line contains one piece of text.",
)
parser.add_argument(
    "--target_schema_path",
    default="/home/ubuntu/project/edc/schemas/rebel_schema.csv",
    help="File containing the target schema to align to.",
)
parser.add_argument("--refinement_iterations", default=0, type=int, help="Number of iteration to run.")
parser.add_argument(
    "--enrich_schema",
    action="store_true",
    help="Whether un-canonicalizable relations should be added to the schema.",
)

# Output setting
parser.add_argument("--output_dir", default="/home/ubuntu/project/edc/output/Mistral-7B-Instruct-v0.2/", help="Directory to output to.")

args = parser.parse_args()
args = vars(args)
print(args)
edc = EDC(**args)

input_text_list = open(args["input_text_file_path"], "r").readlines()
output_kg = edc.extract_kg(input_text_list, args["output_dir"], refinement_iterations=args["refinement_iterations"])
bzhangj13zzz commented 4 weeks ago

@MaskXman May I confirm with you what version of transformers package you are using? I tried to update it to the latest version today and it breaks my code. I suspect if that's the reason. I tested my code to be working fine on transformers=4.39.3

MaskXman commented 4 weeks ago

Of course. image

MaskXman commented 4 weeks ago

Maybe my schema retrivel model is not fine ，Could you please provide --relation_definition_csv_path /output/path/to/tekgen/relation/definitions . Thank you!!!!

bzhangj13zzz commented 4 weeks ago

Thanks a lot for your reply and the information provided, I will carefully look into this issue and I will let you know once it's fixed

bzhangj13zzz commented 4 weeks ago

Possibly related issue with transformers

https://github.com/huggingface/transformers/issues/31251

MaskXman commented 4 weeks ago

Thank you, But I try to your work, My result is not fine, I find the reason ,and I find My schema retireval is not fine, edc_output.txt REBEL/Mistral:

Total scores

Ent_type Correct: 511 Incorrect: 121 Partial: 0 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.07650482640527438 Recall: 0.08195619711299154 F1: 0.07827778659037991

Partial Correct: 507 Incorrect: 0 Partial: 125 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.08816058257322568 Recall: 0.09225983076157292 F1: 0.08940488193848074

Strict Correct: 418 Incorrect: 214 Partial: 0 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.06535711951341618 Recall: 0.06791936286709806 F1: 0.06617072988402008

Exact Correct: 507 Incorrect: 125 Partial: 0 Missed: 5342 Spurious: 651 Possible: 5974 Actual: 1283 Precision: 0.08000430242839401 Recall: 0.08266135722581715 F1: 0.08084201070761549

Scores per tag

Subjects

Ent_type Correct: 236 Incorrect: 49 Partial: 0 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.11512361041977767 Recall: 0.1174713787954206 F1: 0.1157624025219844

Partial Correct: 245 Incorrect: 0 Partial: 40 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.12787041645926664 Recall: 0.13153310104529617 F1: 0.1290277086444334

Strict Correct: 209 Incorrect: 76 Partial: 0 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.10403185664509706 Recall: 0.10403185664509706 F1: 0.10403185664509706

Exact Correct: 245 Incorrect: 40 Partial: 0 Missed: 1699 Spurious: 98 Possible: 1984 Actual: 383 Precision: 0.11937945910071346 Recall: 0.12170233947237431 F1: 0.12015928322548532

Predicates

Ent_type Correct: 116 Incorrect: 3 Partial: 0 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04272045635908254 Recall: 0.04927824788451966 F1: 0.044269977022590265

Partial Correct: 96 Incorrect: 0 Partial: 23 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04207850387541776 Recall: 0.04716276754604281 F1: 0.04330286438300376

Strict Correct: 96 Incorrect: 23 Partial: 0 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04068991127229057 Recall: 0.04355400696864112 F1: 0.041340231584134024

Exact Correct: 96 Incorrect: 23 Partial: 0 Missed: 1882 Spurious: 361 Possible: 2001 Actual: 480 Precision: 0.04068991127229057 Recall: 0.04355400696864112 F1: 0.041340231584134024

Objects

Ent_type Correct: 159 Incorrect: 69 Partial: 0 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.07582545213207234 Recall: 0.07715281234444997 F1: 0.0762734362037498

Partial Correct: 166 Incorrect: 0 Partial: 62 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.0894931143188983 Recall: 0.09619213539074166 F1: 0.09174547867927658

Strict Correct: 113 Incorrect: 115 Partial: 0 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.054878048780487805 Recall: 0.055251368840219015 F1: 0.05501908080305293

Exact Correct: 166 Incorrect: 62 Partial: 0 Missed: 1761 Spurious: 192 Possible: 1989 Actual: 420 Precision: 0.07570101211216194 Recall: 0.08138377302140368 F1: 0.07761738841878214

Full triple scores

Precision: 0.0052173913043478265 Recall: 0.0052173913043478265 F1: 0.0052173913043478265 This is my resulst , so badly, Could you please provide the train realation definition of TeKGEN datasets. Thank you!!!!!

bzhangj13zzz commented 4 weeks ago

@MaskXman I have checked your running parameters again, as written in the README, please use the corresponding few-shot examples when you run experiments on the specific dataset, i.e. when running on REBEL, please use the few-shot examples in the REBEL directory. Please try running the following command:

python run.py \
    --oie_llm mistralai/Mistral-7B-Instruct-v0.2 \
    --oie_few_shot_example_file_path ./few_shot_examples/rebel/oie_few_shot_examples.txt \
    --sd_llm mistralai/Mistral-7B-Instruct-v0.2 \
    --sd_few_shot_example_file_path ./few_shot_examples/rebel/sd_few_shot_examples.txt \
    --sc_llm mistralai/Mistral-7B-Instruct-v0.2 \
    --input_text_file_path ./datasets/rebel.txt \
    --output_dir ./output/rebel \
    --oie_refine_few_shot_example_file_path ./few_shot_examples/rebel/oie_few_shot_refine_examples.txt \
    --ee_llm mistralai/Mistral-7B-Instruct-v0.2 \
    --target_schema ./schemas/rebel_schema.csv \
    --ee_few_shot_example_file_path ./few_shot_examples/rebel/ee_few_shot_examples.txt

I have reproduced the experiments on REBEL with Mistral-7b in all modules, and the results obtained are as follows:

-----------------------------------------------------------------
Total scores
-----------------------------------------------------------------
Ent_type
Correct: 6979 Incorrect: 610 Partial: 0 Missed: 3940
Spurious: 5508 Possible: 11529 Actual: 13097
Precision: 0.4869226853359896 Recall: 0.5031697564521992
F1: 0.4929443971755858
-----------------------------------------------------------------
Partial
Correct: 6593 Incorrect: 0 Partial: 996 Missed: 3940
Spurious: 5508 Possible: 11529 Actual: 13097
Precision: 0.49953614663538326 Recall: 0.5126154125772446
F1: 0.5043149289823227
-----------------------------------------------------------------
Strict
Correct: 6117 Incorrect: 1472 Partial: 0 Missed: 3940
Spurious: 5508 Possible: 11529 Actual: 13097
Precision: 0.43441043083900227 Recall: 0.4422282806252272
F1: 0.4373515010374334
-----------------------------------------------------------------
Exact
Correct: 6593 Incorrect: 996 Partial: 0 Missed: 3940
Spurious: 5508 Possible: 11529 Actual: 13097
Precision: 0.46888984092364683 Recall: 0.47737186477644494
F1: 0.47199396871152594
-----------------------------------------------------------------
Scores per tag
-----------------------------------------------------------------
Subjects
-----------------------------------------------------------------
Ent_type
Correct: 2481 Incorrect: 219 Partial: 0 Missed: 1125
Spurious: 1439 Possible: 3825 Actual: 4139
Precision: 0.5316726385210573 Recall: 0.5389312977099237
F1: 0.5336130584549341
-----------------------------------------------------------------
Partial
Correct: 2352 Incorrect: 0 Partial: 348 Missed: 1125
Spurious: 1439 Possible: 3825 Actual: 4139
Precision: 0.5382619307264891 Recall: 0.5489640130861505
F1: 0.5416035727268006
-----------------------------------------------------------------
Strict
Correct: 2181 Incorrect: 519 Partial: 0 Missed: 1125
Spurious: 1439 Possible: 3825 Actual: 4139
Precision: 0.4741860102819754 Recall: 0.47459105779716465
F1: 0.47433418150975404
-----------------------------------------------------------------
Exact
Correct: 2352 Incorrect: 348 Partial: 0 Missed: 1125
Spurious: 1439 Possible: 3825 Actual: 4139
Precision: 0.5053019681154904 Recall: 0.5116684841875682
F1: 0.5074421422513026
-----------------------------------------------------------------
Predicates
-----------------------------------------------------------------
Ent_type
Correct: 2323 Incorrect: 9 Partial: 0 Missed: 1535
Spurious: 2227 Possible: 3867 Actual: 4559
Precision: 0.4706821761783594 Recall: 0.49574700109051256
F1: 0.4775915251596822
-----------------------------------------------------------------
Partial
Correct: 2063 Incorrect: 0 Partial: 269 Missed: 1535
Spurious: 2227 Possible: 3867 Actual: 4559
Precision: 0.45696413425421056 Recall: 0.47110141766630315
F1: 0.46097522978657113
-----------------------------------------------------------------
Strict
Correct: 2063 Incorrect: 269 Partial: 0 Missed: 1535
Spurious: 2227 Possible: 3867 Actual: 4559
Precision: 0.442264631043257 Recall: 0.4444929116684842
F1: 0.443050319364387
-----------------------------------------------------------------
Exact
Correct: 2063 Incorrect: 269 Partial: 0 Missed: 1535
Spurious: 2227 Possible: 3867 Actual: 4559
Precision: 0.442264631043257 Recall: 0.4444929116684842
F1: 0.443050319364387
-----------------------------------------------------------------
Objects
-----------------------------------------------------------------
Ent_type
Correct: 2175 Incorrect: 382 Partial: 0 Missed: 1280
Spurious: 1842 Possible: 3837 Actual: 4399
Precision: 0.46661577608142496 Recall: 0.4706652126499455
F1: 0.46783403437710963
-----------------------------------------------------------------
Partial
Correct: 2178 Incorrect: 0 Partial: 379 Missed: 1280
Spurious: 1842 Possible: 3837 Actual: 4399
Precision: 0.48948745910577973 Recall: 0.5130861504907307
F1: 0.497275795814509
-----------------------------------------------------------------
Strict
Correct: 1873 Incorrect: 684 Partial: 0 Missed: 1280
Spurious: 1842 Possible: 3837 Actual: 4399
Precision: 0.406034169392948 Recall: 0.40665212649945476
F1: 0.40628135223555073
-----------------------------------------------------------------
Exact
Correct: 2178 Incorrect: 379 Partial: 0 Missed: 1280
Spurious: 1842 Possible: 3837 Actual: 4399
Precision: 0.45287168302435477 Recall: 0.4728462377317339
F1: 0.4595419847328244
-----------------------------------------------------------------
Full triple scores
-----------------------------------------------------------------
Precision: 0.18142068119054924 Recall: 0.18124168967986087
F1: 0.18128260202516108

Please let me know if you still observe a huge discrepancy in the results after re-running the commands I gave.

MaskXman commented 3 weeks ago

Thank you！！ But I re-running the commands you gave , My result is not fine, even if it is improved. I find the LLM work greatly in OIE phrase. This OIE problems is solved, but I find in the SC phrase, abuntant triplets will be delete, because There is no relation definition in the schema, Could you please help me to arrive your result.!!! Thank you

Total scores

Ent_type Correct: 1633 Incorrect: 399 Partial: 0 Missed: 9927 Spurious: 2004 Possible: 11959 Actual: 4036 Precision: 0.11924828924082469 Recall: 0.13152110807000084 F1: 0.12326013333104649

Partial Correct: 1585 Incorrect: 0 Partial: 447 Missed: 9927 Spurious: 2004 Possible: 11959 Actual: 4036 Precision: 0.13778339413758672 Recall: 0.1474060711619806 F1: 0.14088615415438008

Strict Correct: 1277 Incorrect: 755 Partial: 0 Missed: 9927 Spurious: 2004 Possible: 11959 Actual: 4036 Precision: 0.09763652412943281 Recall: 0.10392717923198141 F1: 0.09966040221700834

Exact Correct: 1585 Incorrect: 447 Partial: 0 Missed: 9927 Spurious: 2004 Possible: 11959 Actual: 4036 Precision: 0.12335602317563003 Recall: 0.12970058886953637 F1: 0.125360548743231

Scores per tag

Subjects

Ent_type Correct: 723 Incorrect: 142 Partial: 0 Missed: 3096 Spurious: 253 Possible: 3961 Actual: 1118 Precision: 0.17555569378784108 Recall: 0.17865140582234387 F1: 0.17645193268482645

Partial Correct: 725 Incorrect: 0 Partial: 140 Missed: 3096 Spurious: 253 Possible: 3961 Actual: 1118 Precision: 0.19135253379779382 Recall: 0.19669071908434935 F1: 0.19307199532379926

Strict Correct: 608 Incorrect: 257 Partial: 0 Missed: 3096 Spurious: 253 Possible: 3961 Actual: 1118 Precision: 0.1504188438251638 Recall: 0.15065936800199053 F1: 0.1505065225891302

Exact Correct: 725 Incorrect: 140 Partial: 0 Missed: 3096 Spurious: 253 Possible: 3961 Actual: 1118 Precision: 0.17588123082027038 Recall: 0.17964667827817865 F1: 0.1771466486569746

Predicates

Ent_type Correct: 417 Incorrect: 7 Partial: 0 Missed: 3603 Spurious: 1197 Possible: 4027 Actual: 1621 Precision: 0.07030000355454448 Recall: 0.08907688479721323 F1: 0.07463232904885057

bzhangj13zzz commented 3 weeks ago

Can you clarify what you mean by there is no relation definition? Also can you double check if you have set the target_schema_path to be the schema of REBEL, please make sure all dataset-specific parameters are set to the ones for REBEL.

MaskXman commented 3 weeks ago

Sorry,I find the relation definition in the Target schema , I promise I use rebel all , I will re-check and re-running .Thank you!

bzhangj13zzz commented 3 weeks ago

No problem, please let me know if you have further questions, it will be more helpful if you can attach the parameters you use when reporting the problems, thanks!