DeepGraphLearning / ULTRA

A foundation model for knowledge graph reasoning
MIT License
477 stars 64 forks source link

Something about reimplementation of the results for ultraquery model. #28

Closed leapxcheng closed 2 months ago

leapxcheng commented 2 months ago

Hi,

ULTRA and ULTRAQUERY are both impressive works.

However, when attempting to reimplement the experimental results for ULTRAQUERY (specifically those shown in Table 9) by following the instructions in README.md, I noticed that the performance (as shown following) was lower than expected.

python script/run_query.py -c config/ultraquery/transductive.yaml --dataset FB15kLogicalQuery --epochs 0 --bpe null --gpus [0] --bs 32 --threshold 0.0 --ultra_ckpt null --qe_ckpt /path/to/ckpts/ultraquery.pth

# FB15k237LogicalQuery
hits@1: 0.0088
hits@10: 0.0332
hits@3: 0.0174
mrr: 0.0181

# NELL995LogicalQuery
hits@1: 0.0242
hits@10: 0.0822
hits@3: 0.0471
mrr: 0.0445

I wonder if there might have been some issues during my reimplementation of the experiments. I would greatly appreciate your help with this.

migalkin commented 2 months ago

Did you compile rspmm CUDA kernels and which CUDA toolkit version did you use?

leapxcheng commented 2 months ago

Thank you for your feedback. It seems that the rspmm CUDA kernel is automatically compiled upon the first launch, as described. However, I would like to know how to manually compile the rspmm CUDA kernel and confirm whether it has been compiled successfully. I am currently using CUDA toolkit version 11.8.

leapxcheng commented 2 months ago

I resolved the issue by reinstalling the environment for Ultra, as it appears the rspmm CUDA kernel wasn't successfully compiled before. Thank you for your help!