LLM

[githubgtl]

Describe the question

A clear and concise description of what the question is.

IT RUNS SLOWLY, and the relation task seems a triple extraction, the result on LLM relation extraction can be a record?

Environment (please complete the following information):

• OS: [e.g. mac / window]
• Python Version [e.g. 3.6]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

[guihonghao]

What is a record means?

[zxlzr]

Hi, DeepKE is just a tool to use LLMs for knowledge extraction. Because LLMs have billions of parameters, it is recommend to use GPUs like V100/A100(800) when using LLaMA, ChatGLM or KnowLM.

We provide relational triple extraction as an example for knowledge extraction, you can use your own data (including ner, re, ee etc) for training knowledge extraction models using LLMs with DeepKE-LLM. See https://github.com/zjunlp/DeepKE/blob/main/example/llm/InstructKGC/README.md/#4lora-fine-tuning

Note that for LLMs, knowledge extraction is just a sequence to sequence tasks, so just change the format for different tasks. You should also adjust the hyperparameters.

If you have any problems, please contact us.

[githubgtl]

What is a record means?

as a experiment record about relation extraction in a paper

[guihonghao]

https://github.com/zjunlp/DeepKE/tree/main/example/llm/InstructKGC/kg2instruction#5evaluate Here, we provide a script to extract the corresponding record from the LLM output string and calculate F1

[githubgtl]

https://github.com/zjunlp/DeepKE/tree/main/example/llm/InstructKGC/kg2instruction#5evaluate Here, we provide a script to extract the corresponding record from the LLM output string and calculate F1

I see this script, I have a question, why does my experiment perform badly, only 3.7%, All dataset number is 6000, valid on 3000

[zxlzr]

Could you please provide the details: Which model do you use? LlaMA, KnowLM or ChatGLM? Which data do you use? Your own or the data we provided?

[githubgtl]

Could you please provide the details: Which model do you use? LlaMA, KnowLM or ChatGLM? Which data do you use? Your own or the data we provided?

model:CHATGLM data:chip2022(picked)

[zxlzr]

Could you please provide the details: Which model do you use? LlaMA, KnowLM or ChatGLM? Which data do you use? Your own or the data we provided?

model:CHATGLM data:chip2022(picked)

Based on our previous experiments, the ChatGLM do not perform well on information extraction, maybe you should try KnowLM or Baichuan2 for better performance

You can download KnowLM here(the version for information extraction): https://huggingface.co/zjunlp/knowlm-13b-zhixi

This version has already be combined with the Lora weights.

[githubgtl]

Could you please provide the details: Which model do you use? LlaMA, KnowLM or ChatGLM? Which data do you use? Your own or the data we provided?

model:CHATGLM data:chip2022(picked)

Based on our previous experiments, the ChatGLM do not perform well on information extraction, maybe you should try KnowLM or Baichuan2 for better performance

You can download KnowLM here(the version for information extraction): https://huggingface.co/zjunlp/knowlm-13b-zhixi

This version has already be combined with the Lora weights.

do you have original knowlm model?

You can also try baichuan-7b-chat, if you do not have very large GPUs. We recently only have the 13b version for knowlm.

[zxlzr]

Could you please provide the details: Which model do you use? LlaMA, KnowLM or ChatGLM? Which data do you use? Your own or the data we provided?

model:CHATGLM data:chip2022(picked)

You can follow the steps to use KnowLM https://github.com/zjunlp/KnowLM

[githubgtl]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

[zxlzr]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is better.

[githubgtl]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is

memory >= 60G, wow ,too large

[githubgtl]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is better.

could you add the model code from knowlm and baichuan

[zxlzr]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is better.

could you add the model code from knowlm and baichuan

You can try the code here https://github.com/baichuan-inc/Baichuan-7B https://github.com/baichuan-inc/Baichuan2

[githubgtl]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is better.

could you add the model code from knowlm and baichuan

You can try the code here https://github.com/baichuan-inc/Baichuan-7B https://github.com/baichuan-inc/Baichuan2

I have seen the url "https://huggingface.co/zjunlp/knowlm-13b-ie" in model card, the model checkpoint load small I don't know why

[zxlzr]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is better.

could you add the model code from knowlm and baichuan

You can try the code here https://github.com/baichuan-inc/Baichuan-7B https://github.com/baichuan-inc/Baichuan2

I have seen the url "https://huggingface.co/zjunlp/knowlm-13b-ie" in model card, the model checkpoint load small I don't know why

Have you downloaded the checkpoint? Is it the network issue?

[githubgtl]

hello, sorry to disturb you again, can you tell me what the experiment setting including gpu ,cpu and memary.is baichuan-7b-chat different from baichuan-7b

It is recommended to use GPU >=24G, Memory>=60G for inference. For SFT, the larger the better, we have previously tried two 3090 GPUs, and V100 is better.

could you add the model code from knowlm and baichuan

You can try the code here https://github.com/baichuan-inc/Baichuan-7B https://github.com/baichuan-inc/Baichuan2

I have seen the url "https://huggingface.co/zjunlp/knowlm-13b-ie" in model card, the model checkpoint load small I don't know why

Have you downloaded the checkpoint? Is it the network issue?

yes , I download the checkpoint, and for 20 minutes to wait

[zxlzr]

Maybe you can try to reboot the computer or check the memory usage?

[zxlzr]

have you solved your issue?

[githubgtl]

have you solved your issue?

I am trying to solve it

[githubgtl]

have you solved your issue?

[githubgtl]

have you solved your issue?

i don't know that I download the wrong model or not

[zxlzr]

have you solved your issue?

i don't know that I download the wrong model or not

You can try to download any other 13B model from huggingface and run inference, if it works, it means that you may download the broken knowlm model, if it do not work, it means that there is something wrong with your computer.

[githubgtl]

have you solved your issue?

i don't know that I download the wrong model or not

You can try to download any other 13B model from huggingface and run inference, if it works, it means that you may download the broken knowlm model, if it do not work, it means that there is something wrong with your computer. I find that no mater how many gpus is given,it will tell me oom,maybe single Gpu's memery is small,only 16G

[githubgtl]

have you solved your issue?

i don't know that I download the wrong model or not

You can try to download any other 13B model from huggingface and run inference, if it works, it means that you may download the broken knowlm model, if it do not work, it means that there is something wrong with your computer. I find that no mater how many gpus is given,it will tell me oom,maybe single Gpu's memery is small,only 16G

the knowlm model can't evalute by your infer script, do you have own knowlm script to evalute the valid dataset

[githubgtl]

have you solved your issue?

i don't know that I download the wrong model or not

You can try to download any other 13B model from huggingface and run inference, if it works, it means that you may download the broken knowlm model, if it do not work, it means that there is something wrong with your computer. I find that no mater how many gpus is given,it will tell me oom,maybe single Gpu's memery is small,only 16G

the knowlm model can't evalute by your infer script, do you have own knowlm script to evalute the valid dataset I run the knowlm Model, and the f1 score is also only 3.3%

[githubgtl]

have you solved your issue?

i don't know that I download the wrong model or not

You can try to download any other 13B model from huggingface and run inference, if it works, it means that you may download the broken knowlm model, if it do not work, it means that there is something wrong with your computer. I find that no mater how many gpus is given,it will tell me oom,maybe single Gpu's memery is small,only 16G

the knowlm model can't evalute by your infer script, do you have own knowlm script to evalute the valid dataset I run the knowlm Model, and the f1 score is also only 3.3% I just can't believe it

[zxlzr]

Which dataset do you use? If you are using your own dataset (In your Figure I see a medical instance), you should train the model, the KnowLM (information extraction model) is not designed for domain information extraction.

[githubgtl]

Which dataset do you use? If you are using your own dataset (In your Figure I see a medical instance), you should train the model, the KnowLM (information extraction model) is not designed for domain information extraction.

I have trained the model and then predict the eval dataset

[zxlzr]

You can contact MikeDean2367 (by wechat), and he will help you. The training dataset size and hyperparameters have a big impact, may be the model collapse during training.

[githubgtl]

You can contact MikeDean2367 (by wechat), and he will help you. The training dataset size and hyperparameters have a big impact, may be the model collapse during training.

Ok ,thank you