search

clMathLibraries / clBLAS

a software library containing BLAS functions written in OpenCL
Apache License 2.0
839 stars 240 forks source link

teardown + setup + gemm => "OpenCL error -38 on line 232 of src/library/blas/xgemm.cc" #159

Closed hughperkins closed 8 years ago

hughperkins commented 8 years ago

[Edit: see https://github.com/clMathLibraries/clBLAS/issues/159#issuecomment-150896488 for simple example test case + result.]

"OpenCL error -38 on line 232 of src/library/blas/xgemm.cc"

Full output:

[ RUN      ] testClBlas.colMajor
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
initializing clblas
OpenCL error -38 on line 232 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
OpenCL error -38 on line 232 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
OpenCL error -38 on line 232 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
Segmentation fault

Test case:

TEST(testClBlas, colMajor) {
    EasyCL *cl = EasyCL::createForFirstGpuOtherwiseCpu();

    float A[] = {1, 3,
                 2, 7,
                 9, 5};
    float B[] = {3,
                 -1};

    float C[3];
    transpose(A, 3, 2);
    transpose(B, 2, 1);
    ClBlasInstance clblasInstance;
    CLWrapper *AWrap = cl->wrap(6, A);
    CLWrapper *BWrap = cl->wrap(2, B);
    CLWrapper *CWrap = cl->wrap(3, C);
    AWrap->copyToDevice();
    BWrap->copyToDevice();
    ClBlasHelper::Gemm(
        cl,
        clblasColumnMajor,
        clblasNoTrans, clblasNoTrans,
        3, 2, 1,
        1,
        AWrap, 0,
        BWrap, 0,
        0,
        CWrap, 0
    );
    CWrap->copyToHost();
    transpose(C, 1, 3);
    EXPECT_EQ(0, C[0]);
    EXPECT_EQ(-1, C[1]);
    EXPECT_EQ(22, C[2]);

    delete CWrap;
    delete BWrap;
    delete AWrap;

    delete cl;
}

Device: NVIDIA 940M

By the way, this used to work, using master from around July.

( Basically it succeeds for all tests where rowmajor, and fails for all tests where columnmajor)

hughperkins commented 8 years ago

Just for completeness, please note that after upgrading my NVIDIA drivers to OpenCL 1.2, ie CUDA driver 352.55, errors persist, so probably not because clEnqueueBarrierWithLists was missing:

[ RUN      ] testClBlas.colMajor
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
initializing clblas
OpenCL error -38 on line 232 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
OpenCL error -38 on line 232 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
OpenCL error -38 on line 232 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
Segmentation fault

$ clinfo | grep OpenCL
  Platform Version:              OpenCL 1.2 CUDA 7.5.20
    Execute OpenCL kernels:          Yes
  Device OpenCL C version:           OpenCL C 1.2 
  Version:                   OpenCL 1.2 CUDA
$ clinfo | grep Driver
  Driver version:                352.55
hughperkins commented 8 years ago

gdb trace:

#7  0x00007ffff4f7ea07 in enqueueGemmKernel (clQueue=0x2f960a0, 
    clKernel=0x111fcf0, kernelArgs=<optimized out>, 
    kernelArgSizes=<optimized out>, numKernelArgs=<optimized out>, 
    globalWorkSize=0x7fffffffd860, localWorkSize=0x7fffffffd850, 
    numEventsInWaitList=0, eventWaitList=0x0, clEvent=0x0)
    at /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc:237
#8  0x00007ffff4f7f3c7 in clblasGemm<float> (order=<optimized out>, 
    transA=<optimized out>, transB=<optimized out>, iM=<optimized out>, 
    iN=<optimized out>, iK=<optimized out>, alpha=1, iA=0x2ff86d0, iOffA=0, 
    iLda=3, iB=0x2ff7db0, iOffB=0, iLdb=2, beta=0, C=0x30024c0, iOffC=0, 
    iLdc=3, numCommandQueues=1, commandQueues=0x703310, numEventsInWaitList=0, 
    eventWaitList=0x0, events=0x0)
    at /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc:565

Edit: seems to be the corner kernel:

/******************************************************************************
 * Enqueue Corner kernel
 *****************************************************************************/
  if (needCornerKernel) {
    //printf("enqueueing corner kernel\n");
    size_t globalWorkSize[2] = { 1*workGroupNumRows, 1*workGroupNumCols };
    enqueueGemmKernel( commandQueues[numKernelsEnqueued%numCommandQueues], *cornerClKernel,
      gemmKernelArgs, gemmKernelArgSizes, numGemmKernelArgs,
      globalWorkSize, localWorkSize,
      numEventsInWaitList, eventWaitList,
      &events[numKernelsEnqueued%numCommandQueues] );
    numKernelsEnqueued++;
  }

Edit 2: seems like the rowmajor tests also run enqueue corner kernel, and no other kernels run in fact, but for some reason crashes for colmajor, but not for rowmajor.

Edit 3: added a printf at line 232 of xgemm, and it seems that all of A,B,C are cuasing the error -38:

enqueuekernel i=0
OpenCL error -38 on line 233 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
enqueuekernel i=1
OpenCL error -38 on line 233 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc
enqueuekernel i=2
OpenCL error -38 on line 233 of /home/user/git/DeepCL/clMathLibraries/clBLAS/src/library/blas/xgemm.cc

Edit 4:

Interesting: if I run only colMajor test it passes:

[----------] 1 test from testClBlas
[ RUN      ] testClBlas.colMajor
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
initializing clblas
corner kernel source 
// [snip]
__attribute__((reqd_work_group_size(WG_NUM_COLS,WG_NUM_ROWS,1)))
__kernel void sgemm_Col_NN_B0_ML016_NL016_KX01(
  __global DATA_TYPE_STR const * restrict A,
  __global DATA_TYPE_STR const * restrict B,
  __global DATA_TYPE_STR       *          C,
  DATA_TYPE_STR const alpha,
  DATA_TYPE_STR const beta,
  uint const M,
  uint const N,
  uint const K,
  uint const lda,
  uint const ldb,
  uint const ldc,
  uint const offsetA,
  uint const offsetB,
  uint const offsetC
) {
// [snip]
}

alpha 1.000000 beta 0.000000
M 3 N 1 K 2 lda 3 ldb 2 ldc 3 offA 0 offB 0 offC 0
corner kernel, numgemmkernel args 14
enqueuekernel i=0
enqueuekernel i=1
enqueuekernel i=2
enqueuekernel i=3
enqueuekernel i=4
enqueuekernel i=5
enqueuekernel i=6
enqueuekernel i=7
enqueuekernel i=8
enqueuekernel i=9
enqueuekernel i=10
enqueuekernel i=11
enqueuekernel i=12
enqueuekernel i=13
clblas teardown
[       OK ] testClBlas.colMajor (242 ms)
[----------] 1 test from testClBlas (242 ms total)

Edit 5:

Edit 6: Interestingly, if I run the gemm 3 times, without teardown/setup in between each run, then it all works ok.

So, I reckon there's something going on with setup/teardown.

Since my setup/teardown calls are a bit flaky, might be an issue in my code :-P Checking...

hughperkins commented 8 years ago

Got a repeatable simple example case :-) Basically straight out of the samples, with a loop added:

/* ************************************************************************
 * Copyright 2013 Advanced Micro Devices, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 * ************************************************************************/

#include <sys/types.h>
#include <stdio.h>
#include <string.h>

/* Include CLBLAS header. It automatically includes needed OpenCL header,
 * so we can drop out explicit inclusion of cl.h header.
 */
#include <clBLAS.h>

/* This example uses predefined matrices and their characteristics for
 * simplicity purpose.
 */

#define M  4
#define N  3
#define K  5

static const clblasOrder order = clblasRowMajor;

static const cl_float alpha = 10;

static const clblasTranspose transA = clblasNoTrans;
static const cl_float A[M*K] = {
    11, 12, 13, 14, 15,
    21, 22, 23, 24, 25,
    31, 32, 33, 34, 35,
    41, 42, 43, 44, 45,
};
static const size_t lda = K;        /* i.e. lda = K */

static const clblasTranspose transB = clblasNoTrans;
static const cl_float B[K*N] = {
    11, 12, 13,
    21, 22, 23,
    31, 32, 33,
    41, 42, 43,
    51, 52, 53,
};
static const size_t ldb = N;        /* i.e. ldb = N */

static const cl_float beta = 20;

static cl_float C[M*N] = {
    11, 12, 13,
    21, 22, 23,
    31, 32, 33,
    41, 42, 43,
};
static const size_t ldc = N;        /* i.e. ldc = N */

static cl_float result[M*N];

static const size_t off  = 1;
static const size_t offA = K + 1;   /* K + off */
static const size_t offB = N + 1;   /* N + off */
static const size_t offC = N + 1;   /* N + off */

static void
printResult(const char* str)
{
    size_t i, j, nrows;

    printf("%s:\n", str);

    nrows = (sizeof(result) / sizeof(cl_float)) / ldc;
    for (i = 0; i < nrows; i++) {
        for (j = 0; j < ldc; j++) {
            printf("%d ", (int)result[i * ldc + j]);
        }
        printf("\n");
    }
}

void run() {
    cl_int err;
    cl_platform_id platform = 0;
    cl_device_id device = 0;
    cl_context_properties props[3] = { CL_CONTEXT_PLATFORM, 0, 0 };
    cl_context ctx = 0;
    cl_command_queue queue = 0;
    cl_mem bufA, bufB, bufC;
    cl_event event = NULL;
    int ret = 0;

    /* Setup OpenCL environment. */
    err = clGetPlatformIDs(1, &platform, NULL);
    if (err != CL_SUCCESS) {
        printf( "clGetPlatformIDs() failed with %d\n", err );
        return;
    }
    printf("got platformids\n");

    err = clGetDeviceIDs(platform, CL_DEVICE_TYPE_GPU, 1, &device, NULL);
    if (err != CL_SUCCESS) {
        printf( "clGetDeviceIDs() failed with %d\n", err );
        return;
    }
    printf("got deviceids\n");

    props[1] = (cl_context_properties)platform;
    ctx = clCreateContext(props, 1, &device, NULL, NULL, &err);
    if (err != CL_SUCCESS) {
        printf( "clCreateContext() failed with %d\n", err );
        return;
    }
    printf("created context\n");

    queue = clCreateCommandQueue(ctx, device, 0, &err);
    if (err != CL_SUCCESS) {
        printf( "clCreateCommandQueue() failed with %d\n", err );
        clReleaseContext(ctx);
        return;
    }
    printf("created commandqueue\n");

    /* Setup clblas. */
    err = clblasSetup();
    if (err != CL_SUCCESS) {
        printf("clblasSetup() failed with %d\n", err);
        clReleaseCommandQueue(queue);
        clReleaseContext(ctx);
        return;
    }
    printf("setup blas ok\n");

    /* Prepare OpenCL memory objects and place matrices inside them. */
    bufA = clCreateBuffer(ctx, CL_MEM_READ_ONLY, M * K * sizeof(*A),
                          NULL, &err);
    bufB = clCreateBuffer(ctx, CL_MEM_READ_ONLY, K * N * sizeof(*B),
                          NULL, &err);
    bufC = clCreateBuffer(ctx, CL_MEM_READ_WRITE, M * N * sizeof(*C),
                          NULL, &err);

    err = clEnqueueWriteBuffer(queue, bufA, CL_TRUE, 0,
        M * K * sizeof(*A), A, 0, NULL, NULL);
    err = clEnqueueWriteBuffer(queue, bufB, CL_TRUE, 0,
        K * N * sizeof(*B), B, 0, NULL, NULL);
    err = clEnqueueWriteBuffer(queue, bufC, CL_TRUE, 0,
        M * N * sizeof(*C), C, 0, NULL, NULL);

    /* Call clblas extended function. Perform gemm for the lower right sub-matrices */
    printf("calling sgemm....\n");
    err = clblasSgemm(order, transA, transB, M - off, N - off, K - off,
                         alpha, bufA, offA, lda,
                         bufB, offB, ldb, beta,
                         bufC, offC, ldc,
                         1, &queue, 0, NULL, &event);
    if (err != CL_SUCCESS) {
        printf("clblasSgemmEx() failed with %d\n", err);
        ret = 1;
    }
    else {
        /* Wait for calculations to be finished. */
        err = clWaitForEvents(1, &event);

        /* Fetch results of calculations from GPU memory. */
        err = clEnqueueReadBuffer(queue, bufC, CL_TRUE, 0,
                                  M * N * sizeof(*result),
                                  result, 0, NULL, NULL);

        /* At this point you will get the result of SGEMM placed in 'result' array. */
        puts("");
        printResult("clblasSgemmEx result");
    }

    /* Release OpenCL memory objects. */
    clReleaseMemObject(bufC);
    clReleaseMemObject(bufB);
    clReleaseMemObject(bufA);

    /* Finalize work with clblas. */
    clblasTeardown();

    /* Release OpenCL working objects. */
    clReleaseCommandQueue(queue);
    clReleaseContext(ctx);
}

int
main(void)
{
    for(int i=0; i < 3; i++) {
        printf("i=%i\n", i);
        run();
        printf("finished ok :-)\n");
    }
    return 0;
}

Build like this:

$ gcc -std=c99 -I.. -o test clblas_issue_159.c -l OpenCL -L ~/git/clBLAS/src/build/library -lclBLAS

Run:

$ LD_LIBRARY_PATH=../build/library/ ./test
i=0
got platformids
got deviceids
created context
created commandqueue
setup blas ok
calling sgemm....

clblasSgemmEx result:
11 12 13 
21 35720 36680 
31 50720 52080 
41 65720 67480 
finished ok :-)
i=1
got platformids
got deviceids
created context
created commandqueue
setup blas ok
calling sgemm....
OpenCL error -38 on line 232 of /home/user/git/clBLAS/src/library/blas/xgemm.cc
test: /home/user/git/clBLAS/src/library/blas/xgemm.cc:232: void enqueueGemmKernel(cl_command_queue, cl_kernel, void**, size_t*, unsigned int, const size_t*, const size_t*, cl_uint, _cl_event* const*, _cl_event**): Assertion `false' failed.
Aborted

(Edit: by the way, something I find odd is that clblasSetup is global, ie doesnt take eg a context parameter. I reckon you might consider:

... then ti would be relatively clean, and probably remove this issue fairly painlessly )

hughperkins commented 8 years ago

Guess: maybe it's something to do with the initializations in build/include/AutoGemmIncludes? eg

AutoGemmClKernels.cpp

cl_kernel sgemm_Col_NN_B0_MX096_NX096_KX08_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_ML096_NX096_KX08_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_MX096_NL096_KX08_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_ML096_NL096_KX08_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_MX096_NX096_KX01_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_ML096_NX096_KX01_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_MX096_NL096_KX01_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_ML096_NL096_KX01_clKernel = NULL;
cl_kernel sgemm_Col_NN_B0_MX064_NX064_KX08_clKernel = NULL;
...

AutoGemmKernelBinaries.cpp:

#ifndef KERNEL_SGEMM_COL_NN_B0_MX096_NX096_KX08_BIN_CPP
unsigned char *sgemm_Col_NN_B0_MX096_NX096_KX08_bin = 0;
        size_t sgemm_Col_NN_B0_MX096_NX096_KX08_binSize = 0;
#else
#pragma message("AutoGemmKernelBinaries.cpp: sgemm_Col_NN_B0_MX096_NX096_KX08 was pre-compiled.")
#endif
#ifndef KERNEL_SGEMM_COL_NN_B0_ML096_NX096_KX08_BIN_CPP
unsigned char *sgemm_Col_NN_B0_ML096_NX096_KX08_bin = 0;
        size_t sgemm_Col_NN_B0_ML096_NX096_KX08_binSize = 0;
#else
#pragma message("AutoGemmKernelBinaries.cpp: sgemm_Col_NN_B0_ML096_NX096_KX08 was pre-compiled.")
#endif
#ifndef KERNEL_SGEMM_COL_NN_B0_MX096_NL096_KX08_BIN_CPP
unsigned char *sgemm_Col_NN_B0_MX096_NL096_KX08_bin = 0;
        size_t sgemm_Col_NN_B0_MX096_NL096_KX08_binSize = 0;
#else
...

Edit, so I reckon probably the first one, and this is causing line 104 of xgemm.cc to think the kernel was already built, which it was, but in a previous context :-P

So, we probably want to add a method to eg AutoGemmClKernels.cpp, like initKernels(), which will bascially contain the contents of AutoGemmClKernels.cpp a second time, but without the clkernel prefix on each line.

Edit2: looks like this file is handled by class ClKernelIncludes, in src/library/blas/AutoGemm/Includes.py

hughperkins commented 8 years ago

Fixed :-) Will send a pull request.