search

lyutyuh / ASP

PyTorch implementation and pre-trained models for ASP - Autoregressive Structured Prediction with Language Models, EMNLP 22. https://arxiv.org/pdf/2210.14698.pdf
MIT License
100 stars 15 forks source link

Pretrained Models #4

Open KawshikManikantan opened 1 year ago

KawshikManikantan commented 1 year ago

That is a really interesting idea proposed. Thanks for the release of the codebase. Can you please tell me when the pre-trained models would be released?

Hubotcoder commented 1 year ago

I downloaded them from huggingface, and I guess you should put them in the directory mentioned in each conf, for example: data_dir = ${ASP}/data/conll03_ner/ model_dir = ${ASP}/data/conll03_ner/ log_root = ${ASP}/data/conll03_ner/

KawshikManikantan commented 1 year ago

@Hubotcoder @lyutyuh Huggingface contains pretrained T5 but if I understand right there needs to be additional weights (feedforward networks) for the specific tasks like NER? Please correct me if I am wrong.

Hubotcoder commented 1 year ago

@Hubotcoder @lyutyuh Huggingface contains pretrained T5 but if I understand right there needs to be additional weights (feedforward networks) for the specific tasks like NER? Please correct me if I am wrong.

You are right. Now I get what you mean. Let's wait for the author.

KawshikManikantan commented 1 year ago

@Hubotcoder, tried contacting the authors via mail but no reply even there. How did you advance with this repository? Were you able to train the models?

Thank you

Hubotcoder commented 1 year ago

@Hubotcoder, tried contacting the authors via mail but no reply even there. How did you advance with this repository? Were you able to train the models?

Thank you

@KawshikManikantan Hello. Not yet. I am still asking my boss to buy me a GPU.

Niklss commented 1 year ago

@Hubotcoder, tried contacting the authors via mail but no reply even there. How did you advance with this repository? Were you able to train the models?

Thank you

May be too late, but I've been able to start training t5_large based ere model with flant5_large_conll04 config (except use_amp, I changed it to false. V100 is not working with bf16). It requires 25870MiB of GPU memory.

The last results are this

06/04/2023 21:50:34 - INFO - /home/jovyan/au/ASP/util/runner.py - *******************EPOCH 34*******************
06/04/2023 21:50:37 - INFO - /home/jovyan/au/ASP/util/runner.py - Step 4940: avg loss 0.26; steps/sec 2.45
06/04/2023 21:50:45 - INFO - /home/jovyan/au/ASP/util/runner.py - Step 4960: avg loss 0.18; steps/sec 2.46
06/04/2023 21:50:54 - INFO - /home/jovyan/au/ASP/util/runner.py - Step 4980: avg loss 0.25; steps/sec 2.41
06/04/2023 21:51:02 - INFO - /home/jovyan/au/ASP/util/runner.py - Step 5000: avg loss 0.06; steps/sec 2.52
06/04/2023 21:51:02 - INFO - /home/jovyan/au/ASP/util/runner.py - Dev
06/04/2023 21:51:02 - INFO - /home/jovyan/au/ASP/util/runner.py - Step 5000: evaluating on 231 samples with batch_size 32
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Ent_Precision: 99.7755
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Ent_Recall: 99.5521
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Ent_F1: 99.6637
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_Precision: 98.5465
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_Recall: 98.8338
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_F1: 98.6900
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_p_Precision: 98.5465
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_p_Recall: 98.8338
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_p_F1: 98.6900
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Test
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval max f1: 98.84
06/04/2023 21:51:34 - INFO - /home/jovyan/au/ASP/util/runner.py - Test max f1: 0.00

As you may see, It doesn't performs evaluation on test dataset.

UPD: it doesn't perform test evaluation on the run, but there is a separate method to run the evaluation.

4000 step result (27 epochs)

06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Ent_Precision: 88.8889
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Ent_Recall: 90.4541
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Ent_F1: 89.6647
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_Precision: 72.5806
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_Recall: 74.6445
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_F1: 73.5981
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_p_Precision: 72.5806
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_p_Recall: 74.6445
06/04/2023 22:14:06 - INFO - /home/jovyan/au/ASP/util/runner.py - Eval_Rel_p_F1: 73.5981