search

gunrock / gunrock

Programmable CUDA/C++ GPU Graph Analytics
https://gunrock.github.io/gunrock/
Apache License 2.0
972 stars 200 forks source link

out of memory error of subgraph isomorphism app #833

Open xianglizhi123 opened 3 years ago

xianglizhi123 commented 3 years ago

Summary: Hello. I got out of memory error when I run the subgraph isomorphism app. The query graph I used is the query_sm.mtx in your code directory(triangle graph) and the data graph is gowalla. I got the gowalla graph from the website http://networkrepository.com/soc-gowalla.php. The machine I used is a A100 machine which has 40GB gpu memory.

Reproducibility: Always.

To Reproduce: cd gunrock mkdir build cd build make -j ./sm market soc-gowalla.mtx --pattern-graph-type=market --pattern-graph-file=../../dataset/small/query_sm.mtx --undirected=1 --pattern-undirected=1 --num-runs=1 --quiet=False

GPU: nvidia A100 CPU: Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz OS: Ubuntu 20.04.1 LTS Cuda compilation tools, release 10.1, V10.1.243 NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 gcc-8

Laurawly commented 3 years ago

Hi @xianglizhi123 , you can lower the mem_limit here: https://github.com/gunrock/gunrock/blob/dev/gunrock/app/sm/sm_problem.cuh#L188 to reduce the pre-allocated memory size if you get run out of memory error. Pls lemme know if that helps. Btw which branch did you use?

xianglizhi123 commented 3 years ago

what does this memory limit mean? Does it mean how many bytes it will malloc in GPU?

xianglizhi123 commented 3 years ago

I tried to reduce the memory_limit to 2000000000 which was 3885948780961, but i still get below errors. [/home/lizhi/folder/gunrock/gunrock/util/array_utils.cuh, 481 @ gpu 0] results cudaMalloc failed (CUDA error 2: out of memory) [/home/lizhi/folder/gunrock/gunrock/app/sm/sm_problem.cuh, 195 @ gpu 0] error encountered (CUDA error 2: out of memory) [/home/lizhi/folder/gunrock/gunrock/app/sm/sm_problem.cuh, 502 @ gpu 0] error encountered (CUDA error 2: out of memory) [/home/lizhi/folder/gunrock/gunrock/app/sm/sm_app.cuh, 68 @ gpu 0] error encountered (CUDA error 2: out of memory) [/home/lizhi/folder/gunrock/examples/sm/test_sm.cu, 91 @ gpu 0] error encountered (CUDA error 2: out of memory) My data graph is roadNetCa.mtx and query graph is query_sm.mtx, how should I reduce the memory_limit? I cloned this repo from master branch. Thanks.

Laurawly commented 3 years ago

@xianglizhi123 The memory limit is to ensure that the intermediate results generated in this app needs memory space less than the maximum GPU global memory. Could you try to print out the GPU memory each variable used and locate which line caused the out of memory error? If it's after this line https://github.com/gunrock/gunrock/blob/dev/gunrock/app/sm/sm_problem.cuh#L188, then reduce mem_limit could help. Also I'm not sure if dev branch is currently the same with master. If not, you can also try dev branch.