cogciprocate / ocl

OpenCL for Rust
Other
736 stars 75 forks source link

Driver Bug (Intel, NVIDIA): segfault when building too many programs at once #110

Open rubdos opened 6 years ago

rubdos commented 6 years ago

I had a construction like

loop {
    ProQue::builder().build()
}

due to me benchmarking a simple prototype function. This crashed when looped too often. When ran outside valgrind, the program often worked, but within valgrind, I have consitent crashes.

For future reference, you can find this "bad" code on https://gitlab.com/rubdos/multicore-ocl-project, branch segfault.

When digging deeper, some of the ocl unit tests also seem to be affected:

these tests all crash when ran within valgrind, ending in a segmentation fault.

c0gent commented 6 years ago

Our crack team of valgrinders is on the case...

c0gent commented 6 years ago

If you get a minute could you:

git clone https://github.com/c0gent/ocl-segfault.git 
cd ocl-segfault
cargo run --release

I'm unable to reproduce and I want to see if that segfaults on your system.

c0gent commented 6 years ago

Update: After finally getting the Rust-ocl version to segfault consistently on Intel drivers (but not AMD), I created a an equivalent C++ version, opencl-debug-build-cpp, which also segfaults under the same conditions.

This is definitely an Intel driver issue (possibly NVIDIA too) and not specific to this library. I'll keep this issue open for now until I have time to file bug reports where appropriate.

Since this is problem is restricted to unrealistic use cases such as valgrind and perhaps benchmarks, I don't consider it worth the expense of adding synchronization mechanisms as a work around at present.

It's also unclear whether the size or complexity of the program source code or whether or not the programs are identical are factors.

rubdos commented 6 years ago

Thanks for the heads up, work, and keeping us posted. Appreciate it.

kpp commented 5 years ago

For Intel CPUs this bug was fixed in https://github.com/intel/compute-runtime (replacement for Beignet)