search

chenchens190009 / KG-S2S

COLING 2022: Knowledge Is Flat: A Seq2Seq Generative framework For Various Knowledge Graph Completion
29 stars 6 forks source link

Question about evaluation on WN18RR. #1

Closed estuday closed 1 year ago

estuday commented 2 years ago

hi, nice work. After I reproduced the experiment with your model, the MRR just 0.16. I don't know why it is so different from the results in your paper. Are there any details I need to pay attention to during the experiment?

chenchens190009 commented 2 years ago

hi, estuday,

Thanks for your attention. I am sorry there was an error in the evaluation commandline, but it has been fixed now. Please try this one.

estuday commented 2 years ago

hi, estuday,

Thanks for your attention. I am sorry there was an error in the evaluation commandline, but it has been fixed now. Please try this one.

Thanks, the latest code does solve the problem.

When run the training code, i set the epoch to 1 for convenience. However, the trained model is not saved. I set save_last=True in ModelCheckpoint (the line76 in main.py). It also does not work.

In addition, line 109 in main.py can not load checkpoint path correctly, such as : IsADirectoryError: [Errno 21] Is a directory: '/root/KG-S2S-main'

These errors seem to be caused by the pytorch_lightning version, can you tell me the version you are using, or give some suggestions on the above problems?

Sorry to trouble you again.

chenchens190009 commented 2 years ago

No worries. KG-S2S is evaluated every three epochs by default. So if you configure epoch=1, there would be no model getting saved. You can modify this setting in line: 91 of main.py. In your case, "check_val_every_n_epoch": 1.

Our pytorch_lightning version 1.5.3.

estuday commented 1 year ago

No worries. KG-S2S is evaluated every three epochs by default. So if you configure epoch=1, there would be no model getting saved. You can modify this setting in line: 91 of main.py. In your case, "check_val_every_n_epoch": 1.

Our pytorch_lightning version 1.5.3.

Thanks, i noticed the "check_val_every_n_epoch", however I found that the problem of not saving the model was caused by skip_n_val_epoch. I ignored this parameter at first.

chenchens190009 commented 1 year ago

Yes, that's true.