search

bytedance / byteps

A high performance and generic framework for distributed DNN training
Other
3.63k stars 490 forks source link

undefined symbol: cudaSetupArgument #386

Open harryhan618 opened 3 years ago

harryhan618 commented 3 years ago

Hello! I encounter the error ImportError: /mnt/lustre/hanyibo1/anaconda3/envs/tvm/lib/python3.7/site-packages/byteps-0.2.5-py3.7-linux-x86_64.egg/byteps/torch/c_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: cudaSetupArgument when I start worker process.

Actually it works fine on my another local machine, but not on this cluster machine.