search

dmlc / dgl

Python package built to ease deep learning on graph, on top of existing DL frameworks.
http://dgl.ai
Apache License 2.0
13.51k stars 3.02k forks source link

R-GCN unable to reproduce paper results #3078

Closed TudorAndrei closed 3 years ago

TudorAndrei commented 3 years ago

🐛 Bug

The results of the training process differ from those of the paper.

To Reproduce

Steps to reproduce the behavior:

  1. Train and evaluate model for the WN18 Dataset
  2. Train and evaluate model for the FB15k Dataset
python3 link_predict.py -d wn18 --n-epochs 300 --gpu 0 --eval-protocol filtered --edge-sampler neighbor --n-hidden", "100 --n-bases 10
python3 link_predict.py -d wn18 --n-epochs 300 --gpu 0 --eval-protocol filtered --edge-sampler neighbor --n-hidden 100 --n-bases 10

Expected results

Metric MRR Filtered Hits 1 Hits 3 Hits 10
FB15K .696 .601 .760 .842
DGL* .206 .123 .222 .379
- - - - -
WN18 .819 .697 .929 .964
DGL* .132 .090 .140 .208

Environment

FROM pytorch/pytorch:1.9.0-cuda11.1-cudnn8-runtime

RUN apt-get update && \
apt-get upgrade -y

COPY requirements.txt ./
RUN conda install -c dglteam dgl-cuda11.0
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt

ENV DGLBACKEND='pytorch'
ENV DGL_DOWNLOAD_DIR='/home/stud/user/graph/.dgl'

RUN python -c "import torch"

The image above is build on a GPU cluster

jermainewang commented 3 years ago

There was a similar issue long time ago (https://github.com/dmlc/dgl/issues/697). It seemed that the author's code cannot reproduce paper results either.