Closed Suneal closed 2 years ago
@Suneal I remember encountering similar errors last time I tried.
@Riccorl would it be possible to adapt the model to Huggingface hub: https://huggingface.co/models That would ease training and inference a lot! Thanks!
Hi @Riccorl, I obtained the LDC dataset and used this blog to process and obtain .gold_conll from .gold_skel files.
I was able to train the model using bert_base_span.jsonnet config, and the latest version of transformer_srl.
However, after even after 14 epoch, I do not get 86% F1 for argument labeling. I mention this because I clearly have some examples where the srl_bert_base_conll2012.tar.gz model you provided works but mine fails.
Do you have any idea what might have gone wrong while training using new version of transformer_srl? Would appreciate the help.
This is the result I have right now. Thank you!
{
"best_epoch": 13,
"peak_worker_0_memory_MB": 7488.73828125,
"peak_gpu_0_memory_MB": 10836.53125,
"training_duration": "8:30:54.848019",
"training_start_epoch": 0,
"training_epochs": 13,
"epoch": 13,
"training_precision_role": 0.0,
"training_recall_role": 0.0,
"training_f1_role": 0.0,
"training_precision_frame": 0.999747097492218,
"training_recall_frame": 0.999747097492218,
"training_fscore_frame": 0.999747097492218,
"training_loss": 0.014607782365404236,
"training_worker_0_memory_MB": 7488.73828125,
"training_gpu_0_memory_MB": 10836.53125,
"validation_precision_role": 0.8351303971951395,
"validation_recall_role": 0.8383515741767513,
"validation_f1_role": 0.8367378855684983,
"validation_precision_frame": 0.9546420574188232,
"validation_recall_frame": 0.9546420574188232,
"validation_fscore_frame": 0.9546419978141785,
"validation_loss": 0.44069189061090164,
"best_validation_precision_role": 0.8351303971951395,
"best_validation_recall_role": 0.8383515741767513,
"best_validation_f1_role": 0.8367378855684983,
"best_validation_precision_frame": 0.9546420574188232,
"best_validation_recall_frame": 0.9546420574188232,
"best_validation_fscore_frame": 0.9546419978141785,
"best_validation_loss": 0.44069189061090164
}
Also, I noticed that for the new model, there are issues with handling sentence with repeated verb. Here's an example for the sentence "I can eat an apple but I will not eat an orange." The SRL result completely messes up with the Verb tag.
{ "verbs": [ { "verb": "can", "description": "[ARG0: I] [ARGM-MOD: can] [ARG1: eat an apple] but I will not eat an orange .", "tags": [ "B-ARG0", "B-ARGM-MOD", "B-ARG1", "I-ARG1", "I-ARG1", "O", "O", "O", "O", "O", "O", "O", "O" ], "frame": "can.01", "frame_score": 0.26360154151916504, "lemma": "can" }, { "verb": "eat", "description": "I can eat an apple but I [ARGM-MOD: will] [ARGM-NEG: not] eat an orange .", "tags": [ "O", "O", "O", "O", "O", "O", "O", "B-ARGM-MOD", "B-ARGM-NEG", "O", "O", "O", "O" ], "frame": "eat.01", "frame_score": 0.9999997615814209, "lemma": "eat" }, { "verb": "will", "description": "I can eat an apple but I [go.04: will] not eat an orange .", "tags": [ "O", "O", "O", "O", "O", "O", "O", "B-V", "O", "O", "O", "O", "O" ], "frame": "go.04", "frame_score": 0.3325953483581543, "lemma": "will" }, { "verb": "eat", "description": "I [ARGM-MOD: can] eat an apple but [ARG0: I] [ARGM-MOD: will] [ARGM-NEG: not] eat an orange .", "tags": [ "O", "B-ARGM-MOD", "O", "O", "O", "O", "B-ARG0", "B-ARGM-MOD", "B-ARGM-NEG", "O", "O", "O", "O" ], "frame": "eat.01", "frame_score": 0.9999984502792358, "lemma": "eat" } ], "words": [ "I", "can", "eat", "an", "apple", "but", "I", "will", "not", "eat", "an", "orange", "." ] }
I am not sure if this is a bug for the latest version of tranformer_srl or the newly trained model.
I was able to train the model by reverting to 2.4.6 and making few changes to make the code compatible with the new allennlp.
@Suneal What F1 score did you achieve? Again 86%? Again, would it be possible to adapt the model to (make it compatible with) Huggingface hub using their implementation of transformers: https://huggingface.co/models
@logicReasoner I will have to look into it. Currently, I am using a model for a completely different task. I will definitely let you know if I get to making it compatible with huggingface.
That would be great. Thank you very much
On Fri, Sep 30, 2022, 15:29 Sunil @.***> wrote:
@logicReasoner https://github.com/logicReasoner I will have to look into it. Currently, I am using a model for a completely different task. I will definitely let you know if I get to making it compatible with huggingface.
— Reply to this email directly, view it on GitHub https://github.com/Riccorl/transformer-srl/issues/20#issuecomment-1263510542, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQBYKS37ADGEXRTUUAUT76LWA3MKBANCNFSM6AAAAAAQT6OVCA . You are receiving this because you were mentioned.Message ID: @.***>
The current pre-trained model uses v2.4.6, which requires allennlp<1.3,>=1.2.
Has anybody successfully trained the model for the new version of allennnlp? Would really appreciate the help.