CUDA Invalid Symbol in snn_gpu_module when running hello_world sample

[Runkli]

Below is my output when running the hello_world sample:

*********************************************************************************
********************      Welcome to CARLsim 6.0      ***************************
*********************************************************************************

***************************** Configuring Network ********************************
Starting CARLsim simulation "hello world" in USER mode
Random number seed: 42
Running COBA mode:
  - AMPA decay time            =     5 ms
  - NMDA rise time (disabled)  =     0 ms
  - GABAa decay time           =     6 ms
  - GABAb rise time (disabled) =     0 ms
  - GABAb decay time           =   150 ms

************************** Global Network Configuration *******************************
The number of neurons in the network (numN) = 126
The number of regular neurons in the network (numNReg:numNExcReg:numNInhReg) = 9:9:0
The number of poisson neurons in the network (numNPois:numNExcPois:numInhPois) = 117:117:0
The maximum axonal delay in the network (maxDelay) = 1
CUDA devices Configuration:
  - Number of CUDA devices          =         1
  - CUDA device ID with max GFLOPs  =         0
  + Use CUDA device[0]              = NVIDIA GeForce MX150
  + CUDA Compute Capability (CC)    =       6.1

+ Local Network (0)
|-+ Group List:
  |-+ Local Group input(G:0,L:1): 
    |- Type                       =    EXCIT_POISSON
    |- Num of Neurons             =      117
    |- Start Id                   = (G:9,L:9)
    |- End Id                     = (G:125,L:125)
    |- numPostSynapses            =     1021
    |- numPreSynapses             =        0
    |- Refractory period          =  0.00000
  |-+ Local Group output(G:1,L:0): 
    |- Type                       =    EXCIT
    |- Num of Neurons             =        9
    |- Start Id                   = (G:0,L:0)
    |- End Id                     = (G:8,L:8)
    |- numPostSynapses            =        0
    |- numPreSynapses             =     1021
|-+ Connection List:
  |-+ Local Connection Id 0: input(0) => output(1)
    |- Type                       =    FIXED
    |- Min weight                 =  0.00000
    |- Max weight                 =  0.05000
    |- Initial weight             =  0.05000
    |- Min delay                  =        1
    |- Max delay                  =        1
    |- Radius X                   =     3.00
    |- Radius Y                   =     3.00
    |- Radius Z                   =     1.00
    |- Num of synapses            = 1021
    |- Avg numPreSynapses         =   113.44
    |- Avg numPostSynapses        =     8.73

*****************      Initializing GPU 0 Runtime      *************************
GPU Memory Management: (Total 2002.688 MB)
Data            Size        Total Used  Total Available
Init:           2002.688 MB 35.750 MB   1966.938 MB
Random Gen:     8.000 MB    43.750 MB   1958.938 MB
CUDA error at /home/ilknull/Files/featCARLsim6/carlsim/kernel/src/gpu_module/snn_gpu_module.cu:337 code=13(cudaErrorInvalidSymbol) "cudaMemcpyToSymbol(loadBufferCount, &bufferCnt, sizeof(int), 0, cudaMemcpyHostToDevice)" 

Is this a problem in the source code or the way I'm running my code?

[Runkli]

As loadBufferCount is an int, while cudaMemcpyToSymbol requires a pointer, I changed the argument to &loadBufferCount. However now I am faced with "cudaErrorNoKernelImageForDevice" error, according to NVIDIA it means:

"that there is no kernel image available that is suitable for the device. This can occur when a user specifies code generation options for a particular CUDA source file that do not include the corresponding device configuration. "

So I've either compiled incorrectly for my MX150, or that my GPU may not be supported at all

[Runkli]

MX150 compute capability is 6.1 (according to NVIDIA device query sample code), so I added the gencode flag to the cmake command:

cd .build
make clean
cmake -DCMAKE_INSTALL_PREFIX=/home/ilknull/carlsim6 -DCMAKE_BUILD_TYPE=Release ../. -DCARLSIM_CUDA_GENCODE=-gencode arch=compute_61,code=sm_61
make install

Yet I still get the same error (cudaErrorNoKernelImageForDevice)

[Runkli]

At lines 337:

CUDA_CHECK_ERRORS(cudaMemcpyToSymbol(loadBufferCount, &bufferCnt, sizeof(int), 0, cudaMemcpyHostToDevice));
CUDA_CHECK_ERRORS(cudaMemcpyToSymbol(loadBufferSize, &bufSize, sizeof(int), 0, cudaMemcpyHostToDevice));

loadBuffer variables are being moved from Host to Device, while during declaration they are already "device\" variables. Commenting these out presents me with a "cudaErrorInvalidTexture" error at line 266, cudaBindTexture(__null, groupIdInfo_tex, runtimeData[netId].groupIdInfo, sizeof(int3) * networkConfigs[netId].numGroups)...

[Runkli]

I realize now that the CARLSIM_CUDA_GENCODE flag is used as a variable of its own containing the gencode, such as:

./carlsim/kernel/CMakeLists.txt:                    #-gencode arch=compute_${CARLSIM_CUDA_GENCODE},code=sm_${CARLSIM_CUDA_GENCODE}

and not the entire flag that you would pass to NVCC, as shown in the README

-DCARLSIM_CUDA_GENCODE=-gencode arch=compute_61,code=sm_61

However passing -DCARLSIM_CUDA_GENCODE=61 results in a make install error:

nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified
CMake Error at carlsim-cuda_generated_snn_gpu_module.cu.o.Release.cmake:220 (message):
  Error generating
  /home/ilknull/Files/featCARLsim6/.build/CMakeFiles/carlsim-cuda.dir/carlsim/kernel/src/gpu_module/./carlsim-cuda_generated_snn_gpu_module.cu.o

make[2]: *** [CMakeFiles/carlsim-cuda.dir/build.make:77: CMakeFiles/carlsim-cuda.dir/carlsim/kernel/src/gpu_module/carlsim-cuda_generated_snn_gpu_module.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:316: CMakeFiles/carlsim-cuda.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Though this may be a misunderstanding on my behalf also...

[nmsutton]

@Runkli I don't know if you have already tried this but you could try removing all code containing compute_xx and sm_xx that is not the latest compute_xx that your GPU supports. I think older compute_xx may cause problems with CARLsim running on newer GPUs from Nvidia.

[larsnm]

The user tried to build from an early feature branch of the CARLsim5 repository. With the CARLsim6 repository this issue was resolved. Furthermore, setting CARLSIM_CUDA_GENCODE in cMake is now optional.