search

rapidsai / cugraph

cuGraph - RAPIDS Graph Analytics Library
https://docs.rapids.ai/api/cugraph/stable/
Apache License 2.0
1.74k stars 303 forks source link

[BUG]: MG `ego_graph` Different from SG #4190

Closed nv-rliu closed 7 months ago

nv-rliu commented 8 months ago

Version

24.04

Which installation method(s) does this occur on?

Source

Describe the bug.

Currently, the MG implementation of ego_graph returns a value that differs from the SG implementation when passed multiple n values, aka seeds.

Minimum reproducible example

import cugraph
import cugraph.dask as dcg
from cugraph.datasets import karate
from cugraph.testing.mg_utils import (
    start_dask_client, stop_dask_client
)
import pytest
from cugraph.testing import utils
from cudf.testing.testing import assert_frame_equal

if __name__ == "__main__":
    # start dask
    client, cluster = start_dask_client()

    # Parameters to re-create failure
    data = karate
    directed = True
    seeds = [0, 2]
    radius = 1

    # prep
    data_path = data.get_path()

    ###################################################################
    # SG
    g = data.get_graph(create_using=cugraph.Graph(directed=directed))

    ###################################################################
    # MG
    blocksize = dcg.get_chunksize(data_path)
    ddf = data.get_dask_edgelist()
    dg = cugraph.Graph(directed=directed)
    dg.from_dask_cudf_edgelist(
        ddf,
        source="src",
        destination="dst",
        edge_attr="wgt",
        renumber=True,
        store_transposed=True,
    )

    ###################################################################

    sg_cugraph_ego_graphs = cugraph.batched_ego_graphs(g, seeds=seeds, radius=radius)
    mg_cugraph_ego_graphs = dcg.ego_graph(dg, seeds, radius)

    sg_df, sg_offsets = sg_cugraph_ego_graphs
    mg_df, mg_offsets = mg_cugraph_ego_graphs
    mg_df = mg_df.compute()

    # This Fails
    assert_frame_equals(mg_df, sg_df) 

    # stop dask
    stop_dask_client(client, cluster)

Relevant log output

obj = 'ColumnBase', message = 'values are different (100.0 %)'
left = {'[0, 118, 260]'}, right = {'[0, 142, 260]'}, diff = None

Environment details

<details><summary>Click here to see environment details</summary><pre>

     **git***
     commit 6c73718395a9499e4c01f93b7ab515ad191882c2 (HEAD -> b2404-fix-test-egonet-mg)
     Author: Ralph Liu <ralphl@nvidia.com>
     Date:   Thu Feb 22 09:45:38 2024 -0800

     Use Series.iloc to remove FutureWarning
     **git submodules***

     ***OS Information***
     DGX_NAME="DGX Server"
     DGX_PRETTY_NAME="NVIDIA DGX Server"
     DGX_SWBUILD_DATE="2023-03-27-13-31-04"
     DGX_SWBUILD_VERSION="5.5.0"
     DGX_COMMIT_ID="b2e06e0"
     DGX_PLATFORM="DGX Server for DGX-1"
     DGX_SERIAL_NUMBER="QTFCOU8310012"
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=20.04
     DISTRIB_CODENAME=focal
     DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
     NAME="Ubuntu"
     VERSION="20.04.6 LTS (Focal Fossa)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 20.04.6 LTS"
     VERSION_ID="20.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=focal
     UBUNTU_CODENAME=focal
     Linux dgx16 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Fri Feb 23 14:13:20 2024
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
     | N/A   34C    P0    57W / 300W |  15097MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
     | N/A   33C    P0    43W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
     | N/A   30C    P0    42W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
     | N/A   31C    P0    43W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |
     | N/A   32C    P0    43W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
     | N/A   31C    P0    42W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |
     | N/A   32C    P0    43W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
     | N/A   32C    P0    42W / 300W |      3MiB / 32768MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |    0   N/A  N/A   2854958      C   /opt/conda/bin/python3.10         752MiB |
     |    0   N/A  N/A   3658368      C   /opt/conda/bin/python3.10        3338MiB |
     |    0   N/A  N/A   3839353      C   ...conda3/envs/v5/bin/python    10986MiB |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Byte Order:                      Little Endian
     Address sizes:                   46 bits physical, 48 bits virtual
     CPU(s):                          80
     On-line CPU(s) list:             0-79
     Thread(s) per core:              2
     Core(s) per socket:              20
     Socket(s):                       2
     NUMA node(s):                    2
     Vendor ID:                       GenuineIntel
     CPU family:                      6
     Model:                           79
     Model name:                      Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
     Stepping:                        1
     CPU MHz:                         2776.545
     CPU max MHz:                     3600.0000
     CPU min MHz:                     1200.0000
     BogoMIPS:                        4390.25
     Virtualization:                  VT-x
     L1d cache:                       1.3 MiB
     L1i cache:                       1.3 MiB
     L2 cache:                        10 MiB
     L3 cache:                        100 MiB
     NUMA node0 CPU(s):               0-19,40-59
     NUMA node1 CPU(s):               20-39,60-79
     Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
     Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
     Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Meltdown:          Mitigation; PTI
     Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Retbleed:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d

     ***CMake***
     /home/nfs/ralphl/miniforge3/envs/test-24.04/bin/cmake
     cmake version 3.28.3

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /home/nfs/ralphl/miniforge3/envs/test-24.04/bin/g++
     g++ (conda-forge gcc 11.4.0-5) 11.4.0
     Copyright (C) 2021 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /home/nfs/ralphl/miniforge3/envs/test-24.04/bin/nvcc
/home/nfs/ralphl/miniforge3/envs/test-24.04/bin/nvcc: line 9: /bin/nvcc: No such file or directory

     ***Python***
     /home/nfs/ralphl/miniforge3/envs/test-24.04/bin/python
     Python 3.10.13

     ***Environment Variables***
     PATH                            : /home/nfs/ralphl/miniforge3/envs/test-24.04/bin:/home/nfs/ralphl/miniforge3/condabin:/home/nfs/ralphl/.vscode-server/cli/servers/Stable-903b1e9d8990623e3d7da1df3d33db3e42d80eda/server/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /home/nfs/ralphl/miniforge3/envs/test-24.04
     PYTHON_PATH                     :

     ***conda packages***
     /home/nfs/ralphl/miniforge3/condabin/conda
     # packages in environment at /home/nfs/ralphl/miniforge3/envs/test-24.04:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       2_gnu    conda-forge
     _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
     accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
     aiohttp                   3.9.3           py310h2372a71_0    conda-forge
     aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
     alabaster                 0.7.16             pyhd8ed1ab_0    conda-forge
     anyio                     4.2.0              pyhd8ed1ab_0    conda-forge
     argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
     argon2-cffi-bindings      21.2.0          py310h2372a71_4    conda-forge
     arrow                     1.3.0              pyhd8ed1ab_0    conda-forge
     asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
     async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
     async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
     atk-1.0                   2.38.0               hd4edc92_1    conda-forge
     attrs                     23.2.0             pyh71513ae_0    conda-forge
     aws-c-auth                0.7.15               h70caa3e_0    conda-forge
     aws-c-cal                 0.6.9                h14ec70c_3    conda-forge
     aws-c-common              0.9.12               hd590300_0    conda-forge
     aws-c-compression         0.2.17               h572eabf_8    conda-forge
     aws-c-event-stream        0.4.1                h17cd1f3_5    conda-forge
     aws-c-http                0.8.0                hc6da83f_5    conda-forge
     aws-c-io                  0.14.3               h3c8c088_1    conda-forge
     aws-c-mqtt                0.10.1               h0ef3971_3    conda-forge
     aws-c-s3                  0.5.0                h1b46bed_2    conda-forge
     aws-c-sdkutils            0.1.14               h572eabf_0    conda-forge
     aws-checksums             0.1.17               h572eabf_7    conda-forge
     aws-crt-cpp               0.26.1               h33f84b2_9    conda-forge
     aws-sdk-cpp               1.11.242             h65f022c_0    conda-forge
     babel                     2.14.0             pyhd8ed1ab_0    conda-forge
     beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
     binutils                  2.40                 hdd6e379_0    conda-forge
     binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
     binutils_linux-64         2.40                 hbdbef99_2    conda-forge
     bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
     bokeh                     3.3.4              pyhd8ed1ab_0    conda-forge
     breathe                   4.35.0             pyhd8ed1ab_1    conda-forge
     brotli                    1.1.0                hd590300_1    conda-forge
     brotli-bin                1.1.0                hd590300_1    conda-forge
     brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
     bzip2                     1.0.8                hd590300_5    conda-forge
     c-ares                    1.26.0               hd590300_0    conda-forge
     c-compiler                1.5.2                h0b41bf4_0    conda-forge
     ca-certificates           2024.2.2             hbcca054_0    conda-forge
     cached-property           1.5.2                hd8ed1ab_1    conda-forge
     cached_property           1.5.2              pyha770c72_1    conda-forge
     cachetools                5.3.2              pyhd8ed1ab_0    conda-forge
     cairo                     1.18.0               h3faef2a_0    conda-forge
     certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
     cffi                      1.16.0          py310h2fee648_0    conda-forge
     cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
     charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
     click                     8.1.7           unix_pyh707e725_0    conda-forge
     cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
     cmake                     3.28.3               hcfe8598_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     comm                      0.2.1              pyhd8ed1ab_0    conda-forge
     commonmark                0.9.1                      py_0    conda-forge
     contourpy                 1.2.0           py310hd41b1e2_0    conda-forge
     coverage                  7.4.1           py310h2372a71_0    conda-forge
     cubinlinker               0.3.0           py310hfdf336d_0    rapidsai
     cuda-nvtx                 12.3.101                      0    nvidia
     cuda-profiler-api         11.8.86                       0    nvidia
     cuda-python               11.8.3          py310h70a93da_0    conda-forge
     cuda-version              11.8                 h70ddcb2_2    conda-forge
     cudatoolkit               11.8.0              h4ba93d1_13    conda-forge
     cudf                      24.04.00a406    cuda11_py310_240215_g99ed8b9977_406    rapidsai-nightly
     cugraph                   24.4.0                   pypi_0    pypi
     cupy                      13.0.0          py310h189a05f_3    conda-forge
     cupy-core                 13.0.0          py310h506062a_3    conda-forge
     cxx-compiler              1.5.2                hf52228f_0    conda-forge
     cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
     cython                    3.0.8           py310hc6cd4ac_0    conda-forge
     cytoolz                   0.12.3          py310h2372a71_0    conda-forge
     dask                      2024.1.1           pyhd8ed1ab_0    conda-forge
     dask-core                 2024.1.1           pyhd8ed1ab_0    conda-forge
     dask-cuda                 24.04.00a6      py310_240215_g21376a0_6    rapidsai-nightly
     dask-cudf                 24.04.00a406    cuda11_py310_240215_g99ed8b9977_406    rapidsai-nightly
     debugpy                   1.8.1           py310hc6cd4ac_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
     distributed               2024.1.1           pyhd8ed1ab_0    conda-forge
     dlpack                    0.5                  h9c3ff4c_0    conda-forge
     docutils                  0.19            py310hff52083_1    conda-forge
     doxygen                   1.10.0               h661eb56_0    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
     execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
     executing                 2.0.1              pyhd8ed1ab_0    conda-forge
     expat                     2.5.0                hcb278e6_1    conda-forge
     fastrlock                 0.8.2           py310hc6cd4ac_2    conda-forge
     filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
     fmt                       10.2.1               h00ab1b0_0    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
     fontconfig                2.14.2               h14ed4e7_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fonttools                 4.48.1          py310h2372a71_0    conda-forge
     fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
     freetype                  2.12.1               h267a509_2    conda-forge
     fribidi                   1.0.10               h36c2ea0_0    conda-forge
     frozenlist                1.4.1           py310h2372a71_0    conda-forge
     fsspec                    2024.2.0           pyhca7485f_0    conda-forge
     future                    0.18.3             pyhd8ed1ab_0    conda-forge
     gcc                       11.4.0               h7baecda_2    conda-forge
     gcc_impl_linux-64         11.4.0               h7aa1c59_5    conda-forge
     gcc_linux-64              11.4.0               hfd045f2_2    conda-forge
     gdk-pixbuf                2.42.10              h829c605_4    conda-forge
     gettext                   0.21.1               h27087fc_0    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.1                h0b41bf4_3    conda-forge
     glog                      0.6.0                h6f12383_0    conda-forge
     gmock                     1.14.0               ha770c72_1    conda-forge
     graphite2                 1.3.13            h58526e2_1001    conda-forge
     graphviz                  9.0.0                h78e8752_1    conda-forge
     gtest                     1.14.0               h00ab1b0_1    conda-forge
     gtk2                      2.24.33              h7f000aa_3    conda-forge
     gts                       0.7.6                h977cf35_4    conda-forge
     gxx                       11.4.0               h7baecda_2    conda-forge
     gxx_impl_linux-64         11.4.0               h7aa1c59_5    conda-forge
     gxx_linux-64              11.4.0               hfc1ae95_2    conda-forge
     h11                       0.14.0             pyhd8ed1ab_0    conda-forge
     h2                        4.1.0              pyhd8ed1ab_0    conda-forge
     harfbuzz                  8.3.0                h3d44ed6_0    conda-forge
     hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
     httpcore                  1.0.3              pyhd8ed1ab_0    conda-forge
     httpx                     0.26.0             pyhd8ed1ab_0    conda-forge
     hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
     icu                       73.2                 h59595ed_0    conda-forge
     identify                  2.5.34             pyhd8ed1ab_0    conda-forge
     idna                      3.6                pyhd8ed1ab_0    conda-forge
     imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
     importlib-metadata        7.0.1              pyha770c72_0    conda-forge
     importlib-resources       6.1.1              pyhd8ed1ab_0    conda-forge
     importlib_metadata        7.0.1                hd8ed1ab_0    conda-forge
     importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     ipykernel                 6.29.2             pyhd33586a_0    conda-forge
     ipython                   8.21.0             pyh707e725_0    conda-forge
     isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
     jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
     joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
     json5                     0.9.14             pyhd8ed1ab_0    conda-forge
     jsonpointer               2.4             py310hff52083_3    conda-forge
     jsonschema                4.21.1             pyhd8ed1ab_0    conda-forge
     jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
     jsonschema-with-format-nongpl 4.21.1             pyhd8ed1ab_0    conda-forge
     jupyter-lsp               2.2.2              pyhd8ed1ab_0    conda-forge
     jupyter_client            8.6.0              pyhd8ed1ab_0    conda-forge
     jupyter_core              5.7.1           py310hff52083_0    conda-forge
     jupyter_events            0.9.0              pyhd8ed1ab_0    conda-forge
     jupyter_server            2.12.5             pyhd8ed1ab_0    conda-forge
     jupyter_server_terminals  0.5.2              pyhd8ed1ab_0    conda-forge
     jupyterlab                4.1.1              pyhd8ed1ab_0    conda-forge
     jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
     jupyterlab_server         2.25.2             pyhd8ed1ab_0    conda-forge
     kaleido                   0.2.1                    pypi_0    pypi
     kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     kiwisolver                1.4.5           py310hd41b1e2_1    conda-forge
     krb5                      1.21.2               h659d440_0    conda-forge
     lcms2                     2.16                 hb7c19ff_0    conda-forge
     ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
     libarrow                  14.0.2           he2c5238_5_cpu    conda-forge
     libarrow-acero            14.0.2           h59595ed_5_cpu    conda-forge
     libarrow-dataset          14.0.2           h59595ed_5_cpu    conda-forge
     libarrow-flight           14.0.2           hdc44a87_5_cpu    conda-forge
     libarrow-flight-sql       14.0.2           hfbc7f12_5_cpu    conda-forge
     libarrow-gandiva          14.0.2           hacb8726_5_cpu    conda-forge
     libarrow-substrait        14.0.2           hfbc7f12_5_cpu    conda-forge
     libblas                   3.9.0           21_linux64_openblas    conda-forge
     libbrotlicommon           1.1.0                hd590300_1    conda-forge
     libbrotlidec              1.1.0                hd590300_1    conda-forge
     libbrotlienc              1.1.0                hd590300_1    conda-forge
     libcblas                  3.9.0           21_linux64_openblas    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcublas                 11.11.3.6                     0    nvidia
     libcublas-dev             11.11.3.6                     0    nvidia
     libcudf                   24.04.00a406    cuda11_240215_g99ed8b9977_406    rapidsai-nightly
     libcufile                 1.4.0.31                      0    nvidia
     libcufile-dev             1.4.0.31                      0    nvidia
     libcugraph                24.04.00a43     cuda11_240212_g38a8cdbb3_43    rapidsai-nightly
     libcugraphops             24.04.00a12     cuda11_240214_ge65dfa63_12    rapidsai-nightly
     libcurand                 10.3.0.86                     0    nvidia
     libcurand-dev             10.3.0.86                     0    nvidia
     libcurl                   8.5.0                hca28451_0    conda-forge
     libcusolver               11.4.1.48                     0    nvidia
     libcusolver-dev           11.4.1.48                     0    nvidia
     libcusparse               11.7.5.86                     0    nvidia
     libcusparse-dev           11.7.5.86                     0    nvidia
     libdeflate                1.19                 hd590300_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 hd590300_2    conda-forge
     libevent                  2.1.12               hf998b51_1    conda-forge
     libexpat                  2.5.0                hcb278e6_1    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     11.4.0             h922705a_105    conda-forge
     libgcc-ng                 13.2.0               h807b86a_5    conda-forge
     libgd                     2.3.3                h119a65a_9    conda-forge
     libgfortran-ng            13.2.0               h69a702a_5    conda-forge
     libgfortran5              13.2.0               ha4646dd_5    conda-forge
     libglib                   2.78.3               h783c2da_0    conda-forge
     libgomp                   13.2.0               h807b86a_5    conda-forge
     libgoogle-cloud           2.12.0               hef10d8f_5    conda-forge
     libgrpc                   1.60.1               h74775cd_0    conda-forge
     libhwloc                  2.9.3           default_h554bfaf_1009    conda-forge
     libiconv                  1.17                 hd590300_2    conda-forge
     libidn2                   2.3.7                hd590300_0    conda-forge
     libjpeg-turbo             3.0.0                hd590300_1    conda-forge
     libkvikio                 24.04.00a       cuda11_240212_g9ef242e_6    rapidsai-nightly
     liblapack                 3.9.0           21_linux64_openblas    conda-forge
     libllvm14                 14.0.6               hcd5def8_4    conda-forge
     libllvm15                 15.0.7               hb3ce162_4    conda-forge
     libnghttp2                1.58.0               h47da74e_1    conda-forge
     libnl                     3.9.0                hd590300_0    conda-forge
     libnsl                    2.0.1                hd590300_0    conda-forge
     libnuma                   2.0.16               h0b41bf4_1    conda-forge
     libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
     libparquet                14.0.2           h352af49_5_cpu    conda-forge
     libpng                    1.6.42               h2797004_0    conda-forge
     libprotobuf               4.25.1               hf27288f_1    conda-forge
     libraft                   24.04.00a55     cuda11_240215_gaddb485f_55    rapidsai-nightly
     libraft-headers           24.04.00a55     cuda11_240215_gaddb485f_55    rapidsai-nightly
     libraft-headers-only      24.04.00a55     cuda11_240215_gaddb485f_55    rapidsai-nightly
     libre2-11                 2023.06.02           h7a70373_0    conda-forge
     librmm                    24.04.00a20     cuda11_240215_gce3af2c4_20    rapidsai-nightly
     librsvg                   2.56.3               he3f83f7_1    conda-forge
     libsanitizer              11.4.0               h4dcbe23_5    conda-forge
     libsodium                 1.0.18               h36c2ea0_1    conda-forge
     libsqlite                 3.45.1               h2797004_0    conda-forge
     libssh2                   1.11.0               h0841786_0    conda-forge
     libstdcxx-devel_linux-64  11.4.0             h922705a_105    conda-forge
     libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
     libthrift                 0.19.0               hb90f79a_1    conda-forge
     libtiff                   4.6.0                ha9c0a0a_2    conda-forge
     libunistring              0.9.10               h7f98852_0    conda-forge
     libutf8proc               2.8.0                h166bdaf_0    conda-forge
     libuuid                   2.38.1               h0b41bf4_0    conda-forge
     libuv                     1.47.0               hd590300_0    conda-forge
     libwebp                   1.3.2                h658648e_1    conda-forge
     libwebp-base              1.3.2                hd590300_0    conda-forge
     libwholegraph             24.04.00a18     cuda11_240215_g3abc3a5_18    rapidsai-nightly
     libxcb                    1.15                 h0b41bf4_0    conda-forge
     libxcrypt                 4.4.36               hd590300_1    conda-forge
     libxml2                   2.12.5               h232c23b_0    conda-forge
     libzlib                   1.2.13               hd590300_5    conda-forge
     llvmlite                  0.42.0          py310h1b8f574_1    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lz4                       4.3.3           py310h350c4a5_0    conda-forge
     lz4-c                     1.9.4                hcb278e6_0    conda-forge
     markdown                  3.5.2              pyhd8ed1ab_0    conda-forge
     markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.5           py310h2372a71_0    conda-forge
     matplotlib-base           3.8.2           py310h62c0568_0    conda-forge
     matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
     mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
     mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
     mpi                       1.0                     openmpi    conda-forge
     msgpack-python            1.0.7           py310hd41b1e2_0    conda-forge
     multidict                 6.0.5           py310h2372a71_0    conda-forge
     munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
     nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
     nbconvert                 7.16.0             pyhd8ed1ab_0    conda-forge
     nbconvert-core            7.16.0             pyhd8ed1ab_0    conda-forge
     nbconvert-pandoc          7.16.0             pyhd8ed1ab_0    conda-forge
     nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
     nbsphinx                  0.9.3              pyhd8ed1ab_0    conda-forge
     nccl                      2.20.3.1             h6103f9b_0    conda-forge
     ncurses                   6.4                  h59595ed_2    conda-forge
     nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
     networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
     ninja                     1.11.1               h924138e_0    conda-forge
     nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
     notebook                  7.1.0              pyhd8ed1ab_0    conda-forge
     notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
     numba                     0.59.0          py310h7dc5dd1_1    conda-forge
     numpy                     1.26.4          py310hb13e2d6_0    conda-forge
     numpydoc                  1.6.0              pyhd8ed1ab_0    conda-forge
     nvcc_linux-64             11.8                h41dc85b_23    conda-forge
     nvcomp                    3.0.5                h838ba91_0    conda-forge
     nvtx                      0.2.8           py310h2372a71_1    conda-forge
     openjpeg                  2.5.0                h488ebb8_3    conda-forge
     openmpi                   5.0.1              h4970cb7_101    conda-forge
     openssl                   3.2.1                hd590300_0    conda-forge
     orc                       1.9.2                h7829240_1    conda-forge
     overrides                 7.7.0              pyhd8ed1ab_0    conda-forge
     packaging                 23.2               pyhd8ed1ab_0    conda-forge
     pandas                    2.1.4           py310hcc13569_0    conda-forge
     pandoc                    3.1.11.1             ha770c72_0    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     pango                     1.50.14              ha41ecd1_2    conda-forge
     parso                     0.8.3              pyhd8ed1ab_0    conda-forge
     partd                     1.4.1              pyhd8ed1ab_0    conda-forge
     pathspec                  0.12.1             pyhd8ed1ab_0    conda-forge
     pcre2                     10.42                hcad00b1_0    conda-forge
     pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
     pickleshare               0.7.5                   py_1003    conda-forge
     pillow                    10.2.0          py310h01dd4db_0    conda-forge
     pip                       24.0               pyhd8ed1ab_0    conda-forge
     pixman                    0.43.2               h59595ed_0    conda-forge
     pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
     platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
     plotly                    5.19.0             pyhd8ed1ab_0    conda-forge
     pluggy                    1.4.0              pyhd8ed1ab_0    conda-forge
     pre-commit                3.6.1              pyha770c72_0    conda-forge
     prometheus_client         0.19.0             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
     protobuf                  4.25.1          py310h620c231_0    conda-forge
     psutil                    5.9.8           py310h2372a71_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptxcompiler               0.8.1           py310h70a93da_2    conda-forge
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
     py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
     pyarrow                   14.0.2          py310hf9e7431_5_cpu    conda-forge
     pyarrow-hotfix            0.6                pyhd8ed1ab_0    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pydata-sphinx-theme       0.15.2             pyhd8ed1ab_0    conda-forge
     pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
     pylibcugraph              24.04.00a43     cuda11_py310_240212_g38a8cdbb3_43    rapidsai-nightly
     pylibcugraphops           24.04.00a12     cuda11_py310_240214_ge65dfa63_12    rapidsai-nightly
     pylibraft                 24.04.00a55     cuda11_py310_240215_gaddb485f_55    rapidsai-nightly
     pylibwholegraph           24.04.00a18     cuda11_py310_240212_g3abc3a5_18    rapidsai-nightly
     pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
     pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
     pyproject-metadata        0.7.1              pyhd8ed1ab_0    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     pytest                    8.0.0              pyhd8ed1ab_0    conda-forge
     pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
     pytest-mpl                0.16.1             pyhd8ed1ab_0    conda-forge
     pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge
     python                    3.10.13         hd12c33a_1_cpython    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
     python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
     python-louvain            0.16               pyhd8ed1ab_0    conda-forge
     python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
     python_abi                3.10                    4_cp310    conda-forge
     pytz                      2024.1             pyhd8ed1ab_0    conda-forge
     pyyaml                    6.0.1           py310h2372a71_1    conda-forge
     pyzmq                     25.1.2          py310h795f18f_0    conda-forge
     raft-dask                 24.04.00a55     cuda11_py310_240215_gaddb485f_55    rapidsai-nightly
     rapids-dask-dependency    24.04.00a2                    0    rapidsai-nightly
     rdma-core                 50.0                 hd3aeb46_0    conda-forge
     re2                       2023.06.02           h2873b5e_0    conda-forge
     readline                  8.2                  h8228510_1    conda-forge
     recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
     referencing               0.33.0             pyhd8ed1ab_0    conda-forge
     requests                  2.31.0             pyhd8ed1ab_0    conda-forge
     rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
     rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
     rhash                     1.4.4                hd590300_0    conda-forge
     rich                      13.7.0             pyhd8ed1ab_0    conda-forge
     rmm                       24.04.00a20     cuda11_py310_240215_gce3af2c4_20    rapidsai-nightly
     rpds-py                   0.17.1          py310hcb5633a_0    conda-forge
     s2n                       1.4.3                h06160fa_0    conda-forge
     scikit-build-core         0.8.1              pyh4af843d_0    conda-forge
     scikit-learn              1.4.0           py310h1fdf081_0    conda-forge
     scipy                     1.12.0          py310hb13e2d6_2    conda-forge
     sed                       4.8                  he412f7d_0    conda-forge
     send2trash                1.8.2              pyh41d4057_0    conda-forge
     setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     snappy                    1.1.10               h9fff704_0    conda-forge
     sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
     snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
     spdlog                    1.12.0               hd2e6256_2    conda-forge
     sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
     sphinx-copybutton         0.5.2              pyhd8ed1ab_0    conda-forge
     sphinx-markdown-tables    0.0.17             pyh6c4a22f_0    conda-forge
     sphinxcontrib-applehelp   1.0.8              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-devhelp     1.0.6              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-htmlhelp    2.0.5              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-qthelp      1.0.7              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.10             pyhd8ed1ab_0    conda-forge
     sphinxcontrib-websupport  1.2.7              pyhd8ed1ab_0    conda-forge
     stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
     sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
     tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
     tblib                     3.0.0              pyhd8ed1ab_0    conda-forge
     tenacity                  8.2.3              pyhd8ed1ab_0    conda-forge
     terminado                 0.18.0             pyh0d859eb_0    conda-forge
     threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
     tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
     tk                        8.6.13          noxft_h4845f30_101    conda-forge
     toml                      0.10.2             pyhd8ed1ab_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
     tornado                   6.3.3           py310h2372a71_1    conda-forge
     traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
     types-python-dateutil     2.8.19.20240106    pyhd8ed1ab_0    conda-forge
     typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.9.0              pyha770c72_0    conda-forge
     typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
     tzdata                    2024a                h0c530f3_0    conda-forge
     ucx                       1.15.0               h75e419f_3    conda-forge
     ucx-proc                  1.0.0                       gpu    rapidsai
     ucx-py                    0.37.00a7       py310_240212_ge3d71a4_7    rapidsai-nightly
     ukkonen                   1.0.1           py310hd41b1e2_4    conda-forge
     unicodedata2              15.1.0          py310h2372a71_0    conda-forge
     uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
     urllib3                   2.2.0              pyhd8ed1ab_0    conda-forge
     virtualenv                20.25.0            pyhd8ed1ab_0    conda-forge
     wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
     webcolors                 1.13               pyhd8ed1ab_0    conda-forge
     webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
     websocket-client          1.7.0              pyhd8ed1ab_0    conda-forge
     wget                      1.20.3               ha35d2d1_1    conda-forge
     wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.1.1                hd590300_0    conda-forge
     xorg-libsm                1.2.4                h7391055_0    conda-forge
     xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
     xorg-libxau               1.0.11               hd590300_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
     xorg-libxrender           0.9.11               hd590300_0    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xyzservices               2023.10.1          pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yarl                      1.9.4           py310h2372a71_0    conda-forge
     zeromq                    4.3.5                h59595ed_0    conda-forge
     zict                      3.0.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               hd590300_5    conda-forge
     zstd                      1.5.5                hfc55251_0    conda-forge

</pre></details>

Other/Misc.

No response

Code of Conduct

seunghwak commented 8 months ago

What's the minimum scale (i.e. # GPUs) to reproduce this? Can you reproduce this with 2 GPUs?

nv-rliu commented 8 months ago

What's the minimum scale (i.e. # GPUs) to reproduce this? Can you reproduce this with 2 GPUs?

It's reproducible with 2-GPUs. This was the result from running on a lab machine.


FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/karate.csv-directed:True-seeds:[0, 2]-radius:1] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/karate.csv-directed:True-seeds:[0, 2]-radius:2] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/karate.csv-directed:False-seeds:[0, 2]-radius:1] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/karate.csv-directed:False-seeds:[0, 2]-radius:2] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/dolphins.csv-directed:True-seeds:[0, 2]-radius:1] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/dolphins.csv-directed:True-seeds:[0, 2]-radius:2] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/dolphins.csv-directed:True-seeds:[0, 2]-radius:3] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/dolphins.csv-directed:False-seeds:[0, 2]-radius:1] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/dolphins.csv-directed:False-seeds:[0, 2]-radius:2] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/dolphins.csv-directed:False-seeds:[0, 2]-radius:3] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/email-Eu-core.csv-directed:True-seeds:[0, 2]-radius:1] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/email-Eu-core.csv-directed:True-seeds:[0, 2]-radius:2] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/email-Eu-core.csv-directed:True-seeds:[0, 2]-radius:3] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/email-Eu-core.csv-directed:False-seeds:[0, 2]-radius:1] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/email-Eu-core.csv-directed:False-seeds:[0, 2]-radius:2] - AssertionError: ColumnBase are different
FAILED cugraph/cugraph/tests/sampling/test_egonet_mg.py::test_dask_mg_ego_graphs[graph_file:/home/nfs/ralphl/fix-tests/nv-rliu/datasets/email-Eu-core.csv-directed:False-seeds:[0, 2]-radius:3] - AssertionError: ColumnBase are different
=============================================== 16 failed, 56 passed, 363 warnings in 159.28s (0:02:39) ================================================```
nv-rliu commented 8 months ago

After looking at a trend of 1 Node 8-GPU runs across multiple days, it appears that the failure is transient.

seunghwak commented 8 months ago

You mean this is a heisenbug? Sounds worse but let me try to reproduce this first.

seunghwak commented 8 months ago

How often can you reproduce this? (Say run test_egonet_mg.py 10 times, how many times you see at least one failure?)

I am running this on my local system with 2 GPUs, and I can't reproduce the test failure. Let me try this on a DGX node as well.

seunghwak commented 8 months ago

Never mind, I reproduced this.