search

megagonlabs / zett

:see_no_evil: Code for Zero-shot Triplet Extraction by Template Infilling (Kim et al; IJCNLP-AACL 2023)
https://arxiv.org/abs/2212.10708
BSD 3-Clause "New" or "Revised" License
15 stars 0 forks source link

Reproducing results #2

Open jessganoww opened 8 months ago

jessganoww commented 8 months ago

I'm hoping to reproduce the FewRel results you shared in your paper. It seems that I'm getting false for all data points in the pred_out csv if I run on only a single random seed. Can you confirm whether this behaviour is expected?

Will you also be releasing the scripts or the rd_fold values to reproduce the results from Table 3?

Thank you!

NikitaGautam commented 8 months ago

I am having a similar problem with reproducing the results for the Triple extraction task. I used all the folds for both the fewrel and wikizsl data but the TE results (with the hyperparameters mentioned in the papers/ with different hyperparameters) cannot be reproduced.

bosung commented 8 months ago

Hi, thanks for reporting this. There is definitely a bug in the main branch -- I found it was not updated when we incorporate codes for the final upload. I fixed it and updated to bugfix branch. This is what I've got after running the fixed version on FewRel and n_unseen_rel=10. 

FewRel, n_unseen_rel=10

fold_0: 0.34732314993326413
fold_1: 0.26671597633136096
fold_2: 0.24289855072463767
fold_3: 0.3052043516612761
fold_4: 0.2872695601683844

Thanks for your interests in ZETT!