panic: CUDA_ERROR_LAUNCH_FAILED

tuberosa commented 4 years ago

Hi. I’m using MuMax3 3.10 of this version(https://github.com/mumax/3/tree/jmulkers/prepare3.10) on windows 10.

Until the May 3rd , it was working fine, however though I changed nothing explicitly it started to show the error below when I ran commands like mumax3 test.mx3 on commandline.

I’m wondering GPU driver’s auto-update did something wrong and did downgrading the GPU driver but things didn’t change at all.

Information that may be useful and the whole error messages are at the end of this report.

Any help or comment would be appreciated. Thanks in advance.

・ CUDA toolkit ：ver. 10.0 ・graphics driver ：ver. 441.12 (although the default version in my device was 442.05, I did custom clean install via 411.12 installer.).

・the error I got This error didn’t appear when I set the smaller sample size like gridsize 16x16x2, cellsize 8x8x1.

panic: CUDA_ERROR_LAUNCH_FAILED goroutine 1 [running, locked to thread]: github.com/mumax/3/cuda/cu.Stream.Synchronize(0x0) C:/Users/Usename/go/src/github.com/mumax/3/cuda/cu/stream.go:41 +0x5e github.com/mumax/3/cuda.Sync(...) C:/Users/Usename/go/src/github.com/mumax/3/cuda/init.go:72 github.com/mumax/3/cuda.MemCpyDtoH(0xc01e762000, 0xb09200000, 0x4000000) C:/Users/Usename/go/src/github.com/mumax/3/cuda/slice.go:40 +0x35 github.com/mumax/3/data.Copy(0xc0000647c0, 0xc000096f00) C:/Users/Usename/go/src/github.com/mumax/3/data/slice.go:222 +0x316 github.com/mumax/3/data.(Slice).HostCopy(0xc000096f00, 0x10a8cc0) C:/Users/Usename/go/src/github.com/mumax/3/data/slice.go:203 +0x93 github.com/mumax/3/engine.(magnetization).SetInShape(0x10a64c8, 0xc0001ad110, 0xc0001ad350) C:/Users/Usename/go/src/github.com/mumax/3/engine/magnetization.go:99 +0xe0 reflect.callMethod(0xc000065e40, 0xc0000ada10, 0xc0000ad9f8) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:710 +0x1f3 reflect.methodValueCall(0xc0001ad110, 0xc0001ad350, 0xc0001ad350, 0xc0000a4470, 0xc0000adc48, 0x499072, 0xc002388090, 0xc000065e40, 0xc0000a4470, 0x1000000010, ...) C:/Users/Usename/scoop/apps/go/current/src/reflect/asm_amd64.s:35 +0x49 reflect.Value.call(0x933fe0, 0xc000065e40, 0x13, 0x9db454, 0x4, 0xc0001acb70, 0x2, 0x2, 0x3, 0xc00123a2d0, ...) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:460 +0x8b2 reflect.Value.Call(0x933fe0, 0xc000065e40, 0x13, 0xc0001acb70, 0x2, 0x2, 0x920ee0, 0xc000098120, 0xc000098118) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:321 +0xbb github.com/mumax/3/script.(call).Eval(0xc0001ad740, 0x7a84a9, 0xc000018f70) C:/Users/Usename/go/src/github.com/mumax/3/script/call.go:61 +0x275 github.com/mumax/3/script.(BlockStmt).Eval(0xc0001ad1a0, 0x9201e0, 0x8a2d41) C:/Users/Usename/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68 github.com/mumax/3/script.(forStmt).Eval(0xc000068460, 0x7a84a9, 0xc000018f58) C:/Users/Usename/go/src/github.com/mumax/3/script/for.go:15 +0x57 github.com/mumax/3/script.(BlockStmt).Eval(0xc0001ad0e0, 0x9201e0, 0x8a2d41) C:/Users/Usename/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68 github.com/mumax/3/script.(*forStmt).Eval(0xc000068410, 0x1, 0x1) C:/Users/Usename/go/src/github.com/mumax/3/script/for.go:15 +0x57 github.com/mumax/3/engine.EvalFile(0xc0001acba0) C:/Users/Usename/go/src/github.com/mumax/3/engine/script.go:102 +0x52 main.runScript(0xc0000180c0, 0x8) C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:121 +0x11a main.runFileAndServe(0xc0000180c0, 0x8) C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:92 +0x6e main.main() C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:59 +0x180

JeroenMulkers commented 4 years ago

Is there a reason why you do not want to use the binary provided at the release page?

On Tue, May 5, 2020 at 5:42 AM tuberosa notifications@github.com wrote:

Hi. I’m using MuMax3 3.10 of this version( https://github.com/mumax/3/tree/jmulkers/prepare3.10) on windows 10.

Until the May 3rd , it was working fine, however though I changed nothing explicitly it started to show the error below when I ran commands like mumax3 test.mx3 on commandline.

I’m wondering GPU driver’s auto-update did something wrong and did downgrading the GPU driver but things didn’t change at all.

Information that may be useful and the whole error messages are at the end of this report.

Any help or comment would be appreciated. Thanks in advance.

・ CUDA toolkit ：ver. 10.0 ・graphics driver ：ver. 441.12 (although the default version in my device was 442.05, I did custom clean install via 411.12 installer.).

・the result I got when I executed ‘nvidia-smi’.

+-----------------------------------------------------------------------------+ | NVIDIA-SMI 442.05 Driver Version: 441.12 CUDA Version: 10.2 |

|-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |

|===============================+======================+======================| | 0 GeForce RTX 208... WDDM | 00000000:01:00.0 Off | N/A | | 30% 46C P8 36W / 250W | 310MiB / 11264MiB | 0% Default |

+-------------------------------+----------------------+----------------------+

・the error I got This error didn’t appear when I set the smaller sample size like gridsize 16x16x2, cellsize 8x8x1.

panic: CUDA_ERROR_LAUNCH_FAILED goroutine 1 [running, locked to thread]: github.com/mumax/3/cuda/cu.Stream.Synchronize(0x0) C:/Users/Usename/go/src/github.com/mumax/3/cuda/cu/stream.go:41 +0x5e github.com/mumax/3/cuda.Sync(...) C:/Users/Usename/go/src/ github.com/mumax/3/cuda/init.go:72 github.com/mumax/3/cuda.MemCpyDtoH(0xc01e762000, 0xb09200000, 0x4000000) C:/Users/Usename/go/src/github.com/mumax/3/cuda/slice.go:40 +0x35 github.com/mumax/3/data.Copy(0xc0000647c0, 0xc000096f00) C:/Users/Usename/go/src/github.com/mumax/3/data/slice.go:222 +0x316 github.com/mumax/3/data.(Slice).HostCopy(0xc000096f00, 0x10a8cc0) C:/Users/Usename/go/src/github.com/mumax/3/data/slice.go:203 +0x93 github.com/mumax/3/engine.(magnetization).SetInShape(0x10a64c8, 0xc0001ad110, 0xc0001ad350) C:/Users/Usename/go/src/ github.com/mumax/3/engine/magnetization.go:99 +0xe0 reflect.callMethod(0xc000065e40, 0xc0000ada10, 0xc0000ad9f8) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:710 +0x1f3 reflect.methodValueCall(0xc0001ad110, 0xc0001ad350, 0xc0001ad350, 0xc0000a4470, 0xc0000adc48, 0x499072, 0xc002388090, 0xc000065e40, 0xc0000a4470, 0x1000000010, ...) C:/Users/Usename/scoop/apps/go/current/src/reflect/asm_amd64.s:35 +0x49 reflect.Value.call(0x933fe0, 0xc000065e40, 0x13, 0x9db454, 0x4, 0xc0001acb70, 0x2, 0x2, 0x3, 0xc00123a2d0, ...) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:460 +0x8b2 reflect.Value.Call(0x933fe0, 0xc000065e40, 0x13, 0xc0001acb70, 0x2, 0x2, 0x920ee0, 0xc000098120, 0xc000098118) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:321 +0xbb github.com/mumax/3/script.(call).Eval(0xc0001ad740, 0x7a84a9, 0xc000018f70) C:/Users/Usename/go/src/github.com/mumax/3/script/call.go:61 +0x275 github.com/mumax/3/script.(BlockStmt).Eval(0xc0001ad1a0, 0x9201e0, 0x8a2d41) C:/Users/Usename/go/src/ github.com/mumax/3/script/blockstmt.go:35 +0x68 github.com/mumax/3/script.(forStmt).Eval(0xc000068460, 0x7a84a9, 0xc000018f58) C:/Users/Usename/go/src/github.com/mumax/3/script/for.go:15 +0x57 github.com/mumax/3/script.(BlockStmt).Eval(0xc0001ad0e0, 0x9201e0, 0x8a2d41) C:/Users/Usename/go/src/ github.com/mumax/3/script/blockstmt.go:35 +0x68 github.com/mumax/3/script.(*forStmt).Eval(0xc000068410, 0x1, 0x1) C:/Users/Usename/go/src/github.com/mumax/3/script/for.go:15 +0x57 github.com/mumax/3/engine.EvalFile(0xc0001acba0) C:/Users/Usename/go/src/ github.com/mumax/3/engine/script.go:102 +0x52 main.runScript(0xc0000180c0, 0x8) C:/Users/Usename/go/src/ github.com/mumax/3/cmd/mumax3/main.go:121 +0x11a main.runFileAndServe(0xc0000180c0, 0x8) C:/Users/Usename/go/src/ github.com/mumax/3/cmd/mumax3/main.go:92 +0x6e main.main() C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:59 +0x180

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mumax/3/issues/255, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3LVKOTNIED5NL6RRK77TTRP6DKDANCNFSM4MZHVV6Q .

tuberosa commented 4 years ago

Thanks for your quick response!!

Since I had gotten an error when I tried to use the binary on mumax ver. 3.10 beta some time ago, I stopped to use the binary and decided to recompile by myself.

However, when I downloaded the binary of 3.10 beta for win right now and ran, it didn't show the error which I'd encountered before.

Instead I got errors like below which look essentially the same as the above.

Thanks in advance.

panic: CUDA_ERROR_LAUNCH_FAILED

goroutine 1 [running, locked to thread]:
github.com/mumax/3/cuda/cu.Stream.Synchronize(0x0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cuda/cu/stream.go:41 +0x72
github.com/mumax/3/cuda.Sync()
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cuda/init.go:71 +0x31
github.com/mumax/3/cuda.MemCpyDtoH(0xc00a2cc000, 0xb09200000, 0x4000000)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cuda/slice.go:40 +0x29
github.com/mumax/3/data.Copy(0xc0022ac370, 0xc002257ea0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/data/slice.go:226 +0x39c
github.com/mumax/3/data.(*Slice).HostCopy(0xc002257ea0, 0x472b82)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/data/slice.go:207 +0x8e
github.com/mumax/3/engine.(*magnetization).SetInShape(0xf414b8, 0xc0001528a0, 0xc000153170)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/engine/magnetization.go:94 +0xc7
reflect.callMethod(0xc0001531d0, 0xc0011ab990)
        C:/gosdk/src/reflect/value.go:663 +0x187
reflect.methodValueCall(0xc0001528a0, 0xc000153170, 0xc00005c800, 0xc0021bc140, 0xc0011abc78, 0x4c6af0, 0xc000094510, 0xc0001531d0, 0xc00006b6b0, 0x1000000010, ...)
        C:/gosdk/src/reflect/asm_amd64.s:29 +0x3a
reflect.Value.call(0x8db620, 0xc0001531d0, 0x13, 0x97dd76, 0x4, 0xc0001523c0, 0x2, 0x2, 0xc000153140, 0x8c7140, ...)
        C:/gosdk/src/reflect/value.go:447 +0x450
reflect.Value.Call(0x8db620, 0xc0001531d0, 0x13, 0xc0001523c0, 0x2, 0x2, 0x0, 0x0, 0x0)
        C:/gosdk/src/reflect/value.go:308 +0xab
github.com/mumax/3/script.(*call).Eval(0xc0001539b0, 0x0, 0x0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/call.go:61 +0x29b
github.com/mumax/3/script.(*BlockStmt).Eval(0xc0001530b0, 0x8c6380, 0x84be61)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68
github.com/mumax/3/script.(*forStmt).Eval(0xc0000ae500, 0x757fb9, 0xc00118e620)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/for.go:15 +0x57
github.com/mumax/3/script.(*BlockStmt).Eval(0xc000152ff0, 0x8c6380, 0x84be61)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68
github.com/mumax/3/script.(*forStmt).Eval(0xc0000ae4b0, 0x1, 0x1)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/for.go:15 +0x57
github.com/mumax/3/engine.EvalFile(0xc000152ab0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/engine/script.go:102 +0x52
main.runScript(0xc00008a080, 0x34)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cmd/mumax3/main.go:118 +0x106
main.runFileAndServe(0xc00008a080, 0x34)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cmd/mumax3/main.go:89 +0x6e
main.main()
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cmd/mumax3/main.go:56 +0xe8

tuberosa commented 4 years ago

(I edited the comment just above because I'd misunderstood the word 'binary' before edition. Sorry!)

tuberosa commented 3 years ago

I'm sorry to say that this was a matter of HARDWARE -- GPU.
Since Mumax once worked fine in my PC and the GPU problem occurred a few weeks after I started using it, I was excluding possibility of a bug in hardware, but surprising for me, it WAS.

I found an error when I ran a benchmark on a PC, and replacing GPU worked fine.

mumax / 3

panic: CUDA_ERROR_LAUNCH_FAILED #255