RWTH-ACS / cricket

cricket is a virtualization solution for GPUs
MIT License
137 stars 36 forks source link

error registering fatbin: 222 in cpu-client.c:380 #47

Open Yaegaki1Erika opened 5 months ago

Yaegaki1Erika commented 5 months ago

Hello,the following error occurred when I tried to run the test program cricket.testapp.

image

But when I run it directly without using the server, it can run successfully. Is it a problem with my driver version?

image image

Thank you so much!

n-eiling commented 4 months ago

You are getting CUDA error 222. According to the NVIDIA documentation this is: cudaErrorUnsupportedPtxVersion = 222 This indicates that the provided PTX was compiled with an unsupported toolchain. The most common reason for this, is the PTX was generated by a compiler newer than what is supported by the CUDA driver and PTX JIT compiler.

There seems to be an issue with your setup. You may also have to tweak the arch flag to nvcc in the Makefile that build cricket.testapp.

xvim commented 2 months ago

I've encountered the same issue as well. When I run the program for the first time , it works fine. However, when I run it again with the same parameters, I get an error. no problem running "nbody --benchmark" multiple times

root@desktop:/home/cricket# REMOTE_GPU_ADDRESS=127.0.0.1 LD_PRELOAD=/home/cricket/bin/cricket-client.so  ../cuda/matrix_mul 4096
+08:00:00.000004 INFO:  connection to host "127.0.0.1"
+08:00:00.000145 DEBUG: the command is "matrix_mul_v2"
+08:00:00.000160 DEBUG: using prog=99, vers=1   in cpu-client.c:98
+08:00:00.000171 INFO:  connecting via TCP...
+08:00:00.001383 DEBUG: __cudaRegisterFatBinary(fatCubin=0x637a500f6028)    in cpu-client.c:358
+08:00:00.001790 DEBUG: found new kernel: _Z9matrixMulPfS_S_i (symbol table id: 0x4)    in cpu-elf2.c:975
+08:00:00.001815 DEBUG: cbank_param_size: 0x1c  in cpu-elf2.c:724
+08:00:00.002261 DEBUG: fatbin loaded to 0x637a5148df30
+08:00:00.002278 DEBUG: __cudaRegisterFunction(fatCubinHandle=0x637a5148df30, hostFun=0x637a500f135c, devFunc=_Z9matrixMulPfS_S_i, deviceName=_Z9matrixMulPfS_S_i, thread_limit=-1, tid=[(nil)], bid=[(nil)], bDim=[(nil)], gDim=[(nil)], wSize=(nil))
    in cpu-client.c:322
+08:00:00.002293 DEBUG: request to register known function: "_Z9matrixMulPfS_S_i"   in cpu-client.c:334
Matrix size: 4096 x 4096
Result verification disabled
Using Device 0: "NVIDIA GeForce RTX 4060 Ti"
+08:00:00.881592 DEBUG: cudaLaunchKernel(0x637a500f135c)    in cpu-client-runtime.c:969
+08:00:00.881648 DEBUG: calling kernel "_Z9matrixMulPfS_S_i" (param_size: 28, param_num: 4)
CUDA matrix multiplication completed.
+08:00:01.276752 DEBUG: __cudaUnregisterFatBinary(fatCubinHandle=0x637a5148df30)    in cpu-client.c:394
+08:00:01.276940 INFO:  api-call-cnt: 15
+08:00:01.276957 INFO:  memcpy-cnt: 201326592
root@desktop:/home/cricket# 
root@desktop:/home/cricket# 
root@desktop:/home/cricket# 
root@desktop:/home/cricket# REMOTE_GPU_ADDRESS=127.0.0.1 LD_PRELOAD=/home/cricket/bin/cricket-client.so  ../cuda/matrix_mul 4096
+08:00:00.000004 INFO:  connection to host "127.0.0.1"
+08:00:00.000144 DEBUG: the command is "matrix_mul_v2"
+08:00:00.000160 DEBUG: using prog=99, vers=1   in cpu-client.c:98
+08:00:00.000171 INFO:  connecting via TCP...
+08:00:00.001254 DEBUG: __cudaRegisterFatBinary(fatCubin=0x5fcfc7736028)    in cpu-client.c:358
+08:00:00.001493 DEBUG: found new kernel: _Z9matrixMulPfS_S_i (symbol table id: 0x4)    in cpu-elf2.c:975
+08:00:00.001517 DEBUG: cbank_param_size: 0x1c  in cpu-elf2.c:724
+08:00:00.001700 ERROR: error registering fatbin: 709   in cpu-client.c:380
+08:00:00.001717 DEBUG: __cudaRegisterFunction(fatCubinHandle=(nil), hostFun=0x5fcfc773135c, devFunc=_Z9matrixMulPfS_S_i, deviceName=_Z9matrixMulPfS_S_i, thread_limit=-1, tid=[(nil)], bid=[(nil)], bDim=[(nil)], gDim=[(nil)], wSize=(nil))
    in cpu-client.c:322
+08:00:00.001731 DEBUG: request to register known function: "_Z9matrixMulPfS_S_i"   in cpu-client.c:334
+08:00:00.001843 ERROR: error registering function: -1  in cpu-client.c:344
+08:00:00.001936 INFO:  api-call-cnt: 0
+08:00:00.001951 INFO:  memcpy-cnt: 0
xvim commented 2 months ago

matrix_mul.txt