Unable to reproduce the results in the paper

[hengruizhang98]

Dear authors,

Thanks for sharing the codes! While reproducing your method, I find it difficult to obtain the results presented in the table. Also, the performance is very unstable.

For example, I use the following commands to condense the Cora dataset into a small graph of 140 nodes:

python train_SGDD.py --dataset cora --nlayers=2 --beta 0.1 --r=1.0 --gpu_id=0

The best test accuracy obtained is only about 0.75 at about 400th epoch:

Similarly, I also met the "NAN" loss issue after training a large number of epochs (sometimes it is 400-600th, some times it is after 1000-th).

Another question is about "ogbn-arxiv" dataset. I noticed that the inner_loop is set 0 for ogbn-arxiv dataset. Doesn't it mean that the GNN classifier will not be trained? Also, I failed to obtain a decent performance on ogbn-arxiv dataset using the codes.

My environment is as follows:

absl-py==2.0.0 aiohttp==3.8.6 aiosignal==1.3.1 anndata==0.10.3 anyio @ file:///tmp/build/80754af9/anyio_1644463572971/work/dist appdirs==1.4.4 argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work argon2-cffi-bindings @ file:///tmp/build/80754af9/argon2-cffi-bindings_1644569679365/work array-api-compat==1.4 asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work async-timeout==4.0.3 attrs @ file:///croot/attrs_1668696182826/work backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work beautifulsoup4 @ file:///opt/conda/conda-bld/beautifulsoup4_1650462163268/work bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work brotlipy==0.7.0 cachetools==5.3.1 cdsapi==0.6.1 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi @ file:///croot/cffi_1670423208954/work charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work clean-fid==0.1.35 click==8.1.3 cloudpickle==2.2.1 cmake==3.26.3 comm @ file:///croot/comm_1671231121260/work contourpy==1.0.7 coolgpus==0.23 cryptography @ file:///croot/cryptography_1677533068310/work cycler==0.11.0 daal4py==2023.0.2 dask==2023.3.2 debugpy @ file:///tmp/build/80754af9/debugpy_1637091799509/work decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work deeprobust==0.2.9 defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work dgl==1.0.1+cu117 entrypoints @ file:///tmp/build/80754af9/entrypoints_1649926439650/work exceptiongroup==1.1.3 executing @ file:///opt/conda/conda-bld/executing_1646925071911/work fastjsonschema @ file:///opt/conda/conda-bld/python-fastjsonschema_1661371079312/work filelock==3.11.0 flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core fonttools==4.39.3 frozenlist==1.4.0 fsspec==2023.4.0 gdown==4.7.1 gensim==3.8.3 geomloss==0.2.6 get-annotations==0.1.2 google-auth==2.23.0 google-auth-oauthlib==1.0.0 grpcio==1.58.0 h5py==3.10.0 huggingface-hub==0.13.4 idna @ file:///croot/idna_1666125576474/work imageio==2.33.0 importlib-metadata==6.3.0 importlib-resources==5.12.0 install==1.3.5 ipykernel @ file:///croot/ipykernel_1671488378391/work ipython @ file:///croot/ipython_1676582224036/work ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work ipywidgets==8.0.6 jedi @ file:///tmp/build/80754af9/jedi_1644297102865/work Jinja2 @ file:///croot/jinja2_1666908132255/work joblib @ file:///croot/joblib_1666298844297/work jsonschema @ file:///croot/jsonschema_1676558650973/work jupyter-server @ file:///croot/jupyter_server_1671707632269/work jupyter_client @ file:///croot/jupyter_client_1676329080601/work jupyter_core @ file:///croot/jupyter_core_1676538566912/work jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work jupyterlab-widgets==3.0.7 keopscore==2.1.2 kiwisolver==1.4.4 lazy_loader==0.3 lightning-bolts==0.7.0 lightning-utilities==0.9.0 littleutils==0.2.2 llvmlite==0.41.1 locket==1.0.0 lxml @ file:///opt/conda/conda-bld/lxml_1657545139709/work Markdown==3.4.4 MarkupSafe @ file:///opt/conda/conda-bld/markupsafe_1654597864307/work matplotlib==3.7.1 matplotlib-inline @ file:///opt/conda/conda-bld/matplotlib-inline_1662014470464/work mistune @ file:///tmp/build/80754af9/mistune_1607364877025/work mkl-fft==1.3.1 mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186066731/work mkl-service==2.4.0 multidict==6.0.4 natsort==8.4.0 nb-conda==2.2.1 nb-conda-kernels @ file:///tmp/build/80754af9/nb_conda_kernels_1636035787744/work nbclassic @ file:///croot/nbclassic_1676902904845/work nbclient @ file:///tmp/build/80754af9/nbclient_1650290509967/work nbconvert @ file:///croot/nbconvert_1668450669124/work nbformat @ file:///croot/nbformat_1670352325207/work nest-asyncio @ file:///croot/nest-asyncio_1672387112409/work networkx @ file:///opt/conda/conda-bld/networkx_1657784097507/work notebook @ file:///croot/notebook_1668179881751/work notebook_shim @ file:///croot/notebook-shim_1668160579331/work numba==0.58.1 numpy==1.23.0 oauthlib==3.2.2 ogb==1.3.6 outdated==0.2.2 packaging @ file:///croot/packaging_1671697413597/work pandas==1.5.3 pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work parso @ file:///opt/conda/conda-bld/parso_1641458642106/work partd==1.4.0 pastel==0.2.1 patsy==0.5.3 pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work Pillow==9.4.0 platformdirs @ file:///opt/conda/conda-bld/platformdirs_1662711380096/work poethepoet==0.10.0 pooch @ file:///tmp/build/80754af9/pooch_1623324770023/work POT==0.9.1 prometheus-client @ file:///tmp/abs_d3zeliano1/croots/recipe/prometheus_client_1659455100375/work prompt-toolkit @ file:///croot/prompt-toolkit_1672387306916/work protobuf==4.24.3 psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work pyasn1==0.5.0 pyasn1-modules==0.3.0 pybind11==2.10.4 pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work Pygments @ file:///opt/conda/conda-bld/pygments_1644249106324/work pykeops==2.1.2 pynndescent==0.5.10 pyOpenSSL @ file:///croot/pyopenssl_1677607685877/work pyparsing @ file:///opt/conda/conda-bld/pyparsing_1661452539315/work pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1636110951836/work PySocks @ file:///tmp/build/80754af9/pysocks_1605305812635/work python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work pytorch-lightning==1.9.5 pytz==2023.3 PyYAML==6.0 pyzmq @ file:///opt/conda/conda-bld/pyzmq_1657724186960/work requests @ file:///opt/conda/conda-bld/requests_1657734628632/work requests-oauthlib==1.3.1 rsa==4.9 scanpy==1.9.6 scikit-image==0.22.0 scikit-learn @ file:///croot/scikit-learn_1676911643119/work scikit-learn-intelex==20230228.214108 scipy==1.10.0 scprep==1.2.3 seaborn==0.12.2 Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work session-info==1.0.0 six @ file:///tmp/build/80754af9/six_1644875935023/work smart-open==6.4.0 sniffio @ file:///tmp/build/80754af9/sniffio_1614030464178/work sortedcontainers==2.1.0 soupsieve @ file:///croot/soupsieve_1666296392845/work stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work statsmodels==0.14.0 stdlib-list==0.9.0 tbb==2021.10.0 tensorboard==2.14.0 tensorboard-data-server==0.7.1 tensorboardX==2.6.2.2 terminado @ file:///croot/terminado_1671751832461/work texttable==1.7.0 threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work tifffile==2023.9.26 timm==0.6.13 tinycss2 @ file:///croot/tinycss2_1668168815555/work tomlkit==0.12.2 toolz==0.12.0 torch==1.12.1 torch-cluster @ file:///usr/share/miniconda/envs/test/conda-bld/pytorch-cluster_1656604971222/work torch-geometric @ file:///usr/share/miniconda/envs/test/conda-bld/pyg_1669879312936/work torch-scatter @ file:///usr/share/miniconda/envs/test/conda-bld/pytorch-scatter_1669298921716/work torch-sparse @ file:///usr/share/miniconda/envs/test/conda-bld/pytorch-sparse_1671715073705/work torchaudio==0.12.1 torchcde==0.2.5 torchcfm==1.0.4 torchdiffeq==0.2.3 torchdyn==1.0.6 torchmetrics==1.2.0 torchsde==0.2.6 torchvision==0.13.1 tornado @ file:///opt/conda/conda-bld/tornado_1662061693373/work tqdm @ file:///opt/conda/conda-bld/tqdm_1664392687731/work traitlets @ file:///croot/traitlets_1671143879854/work trampoline==0.1.2 typing_extensions @ file:///croot/typing_extensions_1669924550328/work tzdata==2023.3 umap-learn==0.5.4 urllib3 @ file:///croot/urllib3_1673575502006/work urwid==2.0.1 wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work webencodings==0.5.1 websocket-client @ file:///tmp/build/80754af9/websocket-client_1614803975924/work Werkzeug==2.3.7 widgetsnbextension==4.0.7 xarray==2023.3.0 yarl==1.9.2 zipp==3.15.0

[Suchun-sv]

Thanks for your attention of our work and kindly sharing the environment, maybe there are some inconsistent environment settings. I'll check the code based on your environment as soon as possible, sorry for the inconvenience.

[Suchun-sv]

For the ogbn-arxiv question, since we followed the great GCond work [1]. The default inner 0 is actually used to mimic the one-step algorithm [2]. The recommended setting for ogbn-arxiv is in the range of 3-5, but I cannot ensure if this work in your environment. Thanks for your attention again.

[1] https://github.com/ChandlerBang/GCond [2] Wei Jin, et.al. 2022. Condensing Graphs via One-Step Gradient Matching. KDD '22.

[hengruizhang98]

Thanks for your quick response!

[Suchun-sv]

Thanks for you attention to our work, we find the bug might be the incompatible problem, some function that used to calculating the eigenvalue are removed in the new version of PyTorch. However, we do not catch this exception in this version code.

We fixed these bugs and will soon release the new version code. Hope that will help you.