search

awslabs / dgl-ke

High performance, easy-to-use, and scalable package for learning large-scale knowledge graph embeddings.
https://dglke.dgl.ai/doc/
Apache License 2.0
1.28k stars 196 forks source link

dgl-ke predict errors #232

Closed aridf closed 3 years ago

aridf commented 3 years ago

I am trying to use dglke_predict. I first ran into the error reported in #209 and tried to implement the suggested fix, which was to install from source. To do this I created a new conda environment and installed from source using the instructions on the installation page. Once I had it installed from source, I tried to run my predict command again, which looks like this:

DGLBACKEND=pytorch dglke_predict \
    --model_path ckpts/RotatE_filename_4/ \
    --format 'h_r_t' \
    --data_files results/path/head.list results/path/rel.list results/path/tail.list \
    --raw_data \
    --entity_mfile results/path/entities.tsv \
    --rel_mfile results/path/relations.tsv \
    --topK 5 \
    --exec_mode 'batch_head'

Now I get this error:

Traceback (most recent call last):
  File "/opt/conda/bin/dglke_predict", line 33, in <module>
    sys.exit(load_entry_point('dglke==0.1.0.dev0', 'console_scripts', 'dglke_predict')())
  File "/opt/conda/lib/python3.9/site-packages/dglke-0.1.0.dev0-py3.9.egg/dglke/infer_score.py", line 215, in main
  File "/opt/conda/lib/python3.9/site-packages/dglke-0.1.0.dev0-py3.9.egg/dglke/models/infer.py", line 87, in load_model
KeyError: 'model_name'

Indeed, when I look into the config.json file I see a field "model": "RotatE". I presume this problem has emerged because I trained the model using dglke=0.1.2 and now I'm trying to predict using dglke=0.1.0... I guess this field name was changed?

Then I tried to go back and retrain the using the same command that worked on dlgke-0.1.2:

DGLBACKEND=pytorch dglke_train \
    --data_path results/path --dataset data \
    --data_files entities.tsv relations.tsv all_ctups_30.tsv --format udd_hrt \
    --model_name RotatE \
    --max_step 20000 --batch_size 512 --neg_sample_size 128 --neg_deg_sample --log_interval 100 \
    --hidden_dim 512 --gamma 175 --lr 0.1 -adv --regularization_coef 1.00E-9 \
    --gpu 0 -de

And I get this traceback:

/opt/conda/lib/python3.9/site-packages/dgl/base.py:45: DGLWarning: Recommend creating graphs by `dgl.graph(data)` instead of `dgl.DGLGraph(data)`.
  return warnings.warn(message, category=category, stacklevel=1)
/opt/conda/lib/python3.9/site-packages/dgl/base.py:45: DGLWarning: Keyword arguments ['readonly', 'multigraph', 'sort_csr'] are deprecated in v0.5, and can be safely removed in all cases.
  return warnings.warn(message, category=category, stacklevel=1)
|Train|: 2468118
Traceback (most recent call last):
  File "/opt/conda/bin/dglke_train", line 33, in <module>
    sys.exit(load_entry_point('dglke==0.1.0.dev0', 'console_scripts', 'dglke_train')())
  File "/opt/conda/lib/python3.9/site-packages/dglke-0.1.0.dev0-py3.9.egg/dglke/train.py", line 145, in main
  File "/opt/conda/lib/python3.9/site-packages/dglke-0.1.0.dev0-py3.9.egg/dglke/dataloader/sampler.py", line 410, in create_sampler
  File "/opt/conda/lib/python3.9/site-packages/dgl/contrib/sampling/sampler.py", line 683, in __init__
    self._sampler = _CAPI_CreateUniformEdgeSampler(
  File "dgl/_ffi/_cython/./function.pxi", line 287, in dgl._ffi._cy3.core.FunctionBase.__call__
  File "dgl/_ffi/_cython/./function.pxi", line 232, in dgl._ffi._cy3.core.FuncCall
  File "dgl/_ffi/_cython/./base.pxi", line 155, in dgl._ffi._cy3.core.CALL
dgl._ffi.base.DGLError: [21:13:02] /opt/dgl/include/dgl/packed_func_ext.h:117: Check failed: ObjectTypeChecker<TObjectRef>::Check(sptr.get()): Expected type graph.Graph but get graph.HeteroGraph
Stack trace:
  [bt] (0) /opt/conda/lib/python3.9/site-packages/dgl/libdgl.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4f) [0x7fb7b9afa03f]
  [bt] (1) /opt/conda/lib/python3.9/site-packages/dgl/libdgl.so(dgl::GraphRef dgl::runtime::DGLArgValue::AsObjectRef<dgl::GraphRef>() const+0x264) [0x7fb7b9c7e4a4]
  [bt] (2) /opt/conda/lib/python3.9/site-packages/dgl/libdgl.so(+0x9b1690) [0x7fb7ba293690]
  [bt] (3) /opt/conda/lib/python3.9/site-packages/dgl/libdgl.so(+0x9b2264) [0x7fb7ba294264]
  [bt] (4) /opt/conda/lib/python3.9/site-packages/dgl/libdgl.so(DGLFuncCall+0x48) [0x7fb7ba1edd48]
  [bt] (5) /opt/conda/lib/python3.9/site-packages/dgl/_ffi/_cy3/core.cpython-39-x86_64-linux-gnu.so(+0x15d1e) [0x7fb7b96d9d1e]
  [bt] (6) /opt/conda/lib/python3.9/site-packages/dgl/_ffi/_cy3/core.cpython-39-x86_64-linux-gnu.so(+0x1624b) [0x7fb7b96da24b]
  [bt] (7) /opt/conda/bin/python3(_PyObject_MakeTpCall+0x37f) [0x559226c3769f]
  [bt] (8) /opt/conda/bin/python3(_PyEval_EvalFrameDefault+0x4c6) [0x559226cbd8b6]

Which I'm having trouble deciphering... Please advise.

classicsong commented 3 years ago

Which dgl version you are using? Can you try dlgke-0.1.2 with dgl 0.4.3post2 for both training and prediction?

aridf commented 3 years ago

@classicsong Here's the full output of conda list for the conda environment in which I first received the error. I am running dlgke-0.1.2 and dgl-cuda10.1 0.4.3post2 and this produced the error reported in #209 . So, my first attempt described above used these two, if I'm not mistaken.

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
anyio                     3.0.1            py38h578d9bd_0    conda-forge
argon2-cffi               20.1.0           py38h497a2fe_2    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
blas                      2.108                       mkl    conda-forge
blas-devel                3.9.0                     8_mkl    conda-forge
bleach                    3.3.0              pyh44b312d_0    conda-forge
boto3                     1.17.74            pyhd8ed1ab_0    conda-forge
botocore                  1.20.74            pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
certifi                   2020.12.5        py38h578d9bd_1    conda-forge
cffi                      1.14.5           py38ha65f79e_0    conda-forge
chardet                   4.0.0            py38h578d9bd_1    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cryptography              3.4.7            py38ha5dfef3_0    conda-forge
cudatoolkit               10.1.243             h036e899_8    conda-forge
cycler                    0.10.0                     py_2    conda-forge
dbus                      1.13.18              hb2f20db_0  
decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dgl-cuda10.1              0.4.3post2               py38_0    dglteam
dglke                     0.1.2                    pypi_0    pypi
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
expat                     2.3.0                h9c3ff4c_0    conda-forge
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
glib                      2.68.2               h9c3ff4c_0    conda-forge
glib-tools                2.68.2               h9c3ff4c_0    conda-forge
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               h28cd5cc_2  
hdbscan                   0.8.27           py38h5c078b8_0    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
importlib-metadata        4.0.1            py38h578d9bd_0    conda-forge
ipykernel                 5.3.4            py38h81c977d_1    conda-forge
ipython                   7.23.1           py38hd0cf306_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.18.0           py38h578d9bd_2    conda-forge
jinja2                    3.0.0              pyhd8ed1ab_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9b                   h024ee3a_2  
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter-packaging         0.7.12             pyhd8ed1ab_0    conda-forge
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py38h578d9bd_0    conda-forge
jupyter_server            1.7.0            py38h578d9bd_1    conda-forge
jupyterlab                3.0.12             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.5.2              pyhd8ed1ab_0    conda-forge
kiwisolver                1.3.1            py38h1fd1430_1    conda-forge
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libblas                   3.9.0                     8_mkl    conda-forge
libcblas                  3.9.0                     8_mkl    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgfortran-ng            9.3.0               hff62375_19    conda-forge
libgfortran5              9.3.0               hff62375_19    conda-forge
libglib                   2.68.2               h3e27bee_0    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0                     8_mkl    conda-forge
liblapacke                3.9.0                     8_mkl    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libtiff                   4.1.0                h2733197_1  
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.41.0               h7f98852_0    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxml2                   2.9.9                h13577e0_2    conda-forge
llvm-openmp               11.1.0               h4bd325d_1    conda-forge
llvmlite                  0.36.0           py38h4630a5e_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markupsafe                2.0.0            py38h497a2fe_0    conda-forge
matplotlib                3.3.4            py38h578d9bd_0    conda-forge
matplotlib-base           3.3.4            py38h0efea84_0    conda-forge
matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
mistune                   0.8.4           py38h497a2fe_1003    conda-forge
mkl                       2020.4             h726a3e6_304    conda-forge
mkl-devel                 2020.4             ha770c72_305    conda-forge
mkl-include               2020.4             h726a3e6_304    conda-forge
nbclassic                 0.2.8              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.0.7            py38h578d9bd_3    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
networkx                  2.5                        py_0    conda-forge
ninja                     1.10.2               h4bd325d_0    conda-forge
notebook                  6.4.0              pyha770c72_0    conda-forge
numba                     0.53.1           py38h0e12cce_0    conda-forge
numpy                     1.20.2           py38h9894fe3_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.1.5            py38h51da96c_0    conda-forge
pandoc                    2.13                 h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.2.0            py38he98fc37_0  
pip                       20.3.3             pyhd8ed1ab_0    conda-forge
prometheus_client         0.10.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.18             pyha770c72_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
pymongo                   3.11.2           py38h2531618_0  
pynndescent               0.5.2              pyh44b312d_0    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyqt                      5.9.2            py38h05f1152_4  
pyrsistent                0.17.3           py38h497a2fe_2    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
python                    3.8.6           hffdb5ce_5_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytorch                   1.7.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyyaml                    5.3.1            py38h497a2fe_2    conda-forge
pyzmq                     22.0.3           py38h2035c66_1    conda-forge
qt                        5.9.7                h5867ecd_1  
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
s3transfer                0.4.2              pyhd8ed1ab_0    conda-forge
scikit-learn              0.23.2           py38h5d63f67_3    conda-forge
scipy                     1.6.2                    pypi_0    pypi
seaborn                   0.11.1               hd8ed1ab_1    conda-forge
seaborn-base              0.11.1             pyhd8ed1ab_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                49.6.0           py38h578d9bd_3    conda-forge
sip                       4.19.13          py38he6710b0_0  
six                       1.16.0             pyh6c4a22f_0    conda-forge
slack-log-handler         0.3.0                    pypi_0    pypi
sniffio                   1.2.0            py38h578d9bd_1    conda-forge
sqlite                    3.35.5               h74cdb3f_0    conda-forge
statsmodels               0.12.2           py38h5c078b8_0    conda-forge
tbb                       2020.2               h4bd325d_4    conda-forge
terminado                 0.9.5            py38h578d9bd_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
torchaudio                0.7.2                      py38    pytorch
torchvision               0.8.2                py38_cu101    pytorch
tornado                   6.1              py38h497a2fe_1    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
umap-learn                0.5.1            py38h578d9bd_0    conda-forge
urllib3                   1.26.4             pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.57.0           py38h578d9bd_4    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge
aridf commented 3 years ago

Possibly related: What should head.list, tail.list, and rel.list look like? Currently these are text files with one entry per line.

aridf commented 3 years ago

@classicsong What version of dgl should I have installed if I'm using the version built from source?

classicsong commented 3 years ago

dgl 0.4.3post2

aridf commented 3 years ago

How can I install this? Per the dgl-ke docs I tried pip3 install dgl==0.4.3 and received the following:

ERROR: Could not find a version that satisfies the requirement dgl==0.4.3
ERROR: No matching distribution found for dgl==0.4.3
classicsong commented 3 years ago

pip install dgl==0.4.3post2 or pip install dgl-cu102==0.4.3post2 (cuxxx depends on your cuda version.

aridf commented 3 years ago

After upgrading pip to version 21.2.4 I now get the following, more informative error when trying to install dgl:

ERROR: Could not find a version that satisfies the requirement dgl==0.4.3 (from versions: 0.1.0, 0.1.2, 0.1.3, 0.6.0, 0.6.0.post1, 0.6.1, 0.7a210406, 0.7a210407, 0.7a210408, 0.7a210409, 0.7a210410, 0.7a210412, 0.7a210413, 0.7a210414, 0.7a210415, 0.7a210416, 0.7a210420, 0.7a210421, 0.7a210422, 0.7a210423, 0.7a210424, 0.7a210425, 0.7a210426, 0.7a210427, 0.7a210429, 0.7a210501, 0.7a210503, 0.7a210506, 0.7a210507, 0.7a210508, 0.7a210511, 0.7a210512, 0.7a210513, 0.7a210514, 0.7a210515, 0.7a210517, 0.7a210518, 0.7a210519, 0.7a210520, 0.7a210525, 0.7a210527)
ERROR: No matching distribution found for dgl==0.4.3

It looks like 0.4.3 is not available in pip?

classicsong commented 3 years ago

I can install it through sudo pip3 install dgl==0.4.3post2 locally.

aridf commented 3 years ago

We resolved this by installing dgl==0.4.3.post2 and installing dgl-ke from source, while pinning python version to 3.8. The previous errors were emerging because in my new environment I was using version 3.9