Open ankush9812 opened 2 years ago
Hi Ankush, thanks for your interest.
We will be adding the QA finetuning clean code soon. However, if you want to do it yourself earlier (and are ok with modifying the code), you just need to follow these steps:
main_accelerate.py
to model = T5ForConditionalGeneration.from_pretrained('t5-base')
train.txt
of a new dataset folder e.g. mydataset_qa
. I would also suggest you add some lines from the link prediction style dataset to this file as well, maybe the same number as the number of QA lines (refer to sec 3.4 regularisation scheme). Then run training with that dataset. Also, make sure to specify load_checkpoint
command line arg of the link prediction model so that the pretrained model is loaded for finetuning.apoorv-dump
branch eval_accelerate.py
for reference.What is the link prediction style dataset? I have created the 5000.pt file for the triplets of my constructed KG. I have the train.txt file with question-answer pairs in the mydataset_qa folder. What will be the next steps for fine-tuning and testing? I'm unclear about it.
Could you please tell me, roughly,
Based on this I can suggest you the best next steps
Is this information sufficient? What should be my next steps for fine-tuning?
Since the number of QA pairs is relatively small, I would recommend the following:
predict tail: obama | president of\tUnited States
predict head: United states | president of\tobama
predict tail: ....
.
.
predict answer: what is the currency in china?\tYuan
predict answer: ...
So the number of lines in train.txt would be: no. of train triples x 2 + no. of qa pairs
Thank you for the reply. I have a doubt in KGT5, pl correct me if I'm wrong. According to me, training the LM model with small KG or domain-specific will not work very well compared to finetuning pre-trained language models(results for my KG are shown with Bert and Gpt-3 in this paper ). Can we develop a pipeline in which we use pre-trained LM and can somehow train it with triplets and fine-tune it with QA pairs? I think, currently, there is a lot of work in domain-specific KG construction. Can we train the LM model by masking the text from which KG is constructed? Then, use triplets to train it better and finally fine-tune it with QA pairs. Something similar is done in QA-GNN and GreaseLM but not exactly what I had proposed. Pl, give your comments on this.
Yes, starting with pretrained LM should be extremely helpful in such a dataset. As mentioned in the first reply, you can modify the code to do this. Here are the complete steps again:
main_accelerate.py
to model = T5ForConditionalGeneration.from_pretrained('t5-base')
so that whenever training is started, it uses the pretrained LM. Choose LM size based on your capacity - if you have a smaller dataset, you can probably afford a larger model.predict tail: obama | president of\tUnited States
predict head: United states | president of\tobama
predict tail: ....
.
.
predict answer: what is the currency in china?\tYuan
predict answer: ...
So the number of lines in train.txt
would be: no. of train triples x 2 + no. of qa pairs. Place this train.txt in a new dataset folder (e.g. mydataset)
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node 4 --use_env ./main_accelerate.py \
--save_prefix mydataset_t5base \
--model_size base --dataset mydataset \
--batch_size 8 --save_steps 5000 \
--loss_steps 500 --epochs 10
Please choose largest batch size that doesn't cause OOM
Can we train the LM model by masking the text from which KG is constructed
I have not addressed this in the above comment. This can probably be done, but it is out of the scope of this repo. Please let me know if you are able to do this and it improves results
Thank you for the answer.
Hi Apoorv Great work with KGT5 model.
I basically want to implement the entire pipeline(KGT5) for my constructed KG. I published a paper(Knowledge Graph – Deep Learning: A Case Study in Question Answering in Aviation Safety Domain) in LREC 2022 where I contributed the Aviation KG and showed results with the KG+DL QA system. Our combined QA system performance was better than individually constructed DLQA and KGQA systems. In your paper, I'll work with the training of triplets but need help in fine-tuning the model as your other branch code is not clean so not really understandable. If README can be provided it would be great.