search

uma-pi1 / kge

LibKGE - A knowledge graph embedding library for reproducible research
MIT License
765 stars 124 forks source link

Dealing with tie-handling errors #242

Closed BugsBuggy closed 2 years ago

BugsBuggy commented 2 years ago

Hello together, I saw the previous history of tie-handling errors during evaluation in #224. I am still getting those errors during evaluation for example when trying to reproduce the exact same config from iclr20 random search with DistMult on FB15k-237 with negative sampling. I tried to increase the tolerance levels to atol: 1e-03 and rtol: 1e-02 but that did not help. I could not figure out smaller values for rtol and atol yet which do not result in tie-handling errors over many runs.

Are there any best practices on how to set these parameters or other ways to avoid these issues?

AdrianKs commented 2 years ago

Issues with tie-handling are treated as a warning now and will not abort the train/eval job. Additionally with #222 we changed the calculation of the scores of the positive triples so that we hopefully don't see the discrepancies between ties anymore. You can try again using the default settings for the tolerances.

AdrianKs commented 2 years ago

I will close this issue for now, please reopen if this issue still occurs.