[BUG] setting cuda device outside of MagmaDNN

MagmaDNN / magmadnn

MagmaDNN: a simple deep learning framework in c++

MIT License

45 stars 7 forks source link

[BUG] setting cuda device outside of MagmaDNN #15

Closed Dando18 closed 5 years ago

Dando18 commented 5 years ago

Describe the bug Calling cudaSetDevice outside of MagmaDNN causes segfaults within MagmaDNN code.

To Reproduce See

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
cudaSetDevice(rank % n_gpus);
...
magmadnn code here
...

Running the above code before a MagmaDNN gpu program will cause it to segfault.

Expected behavior no segfault

Environment:

OS: 18.10
CUDA Version: 10.1
CBLAS: IntelMKL
Magma Version: 2.5.2
CuDNN Version: 7.6
MagmaDNN Version: 1.0

Dando18 commented 5 years ago

This may be good reason to switch MagmaDNN GPU operations to use cuda streams. Magma (v2) and CuDNN support stream use so it wouldn't be too difficult to change.

Dando18 commented 5 years ago

Fixed in bb2d1211b1cc2dbdc5caf20348875ca536e2643a.

Currently fixed in dev and will make its way to master soon (as soon as tests pass and the v1.1 milestone is reached).