sgkit-dev / sgkit

Scalable genetics toolkit
https://sgkit-dev.github.io/sgkit
Apache License 2.0
233 stars 32 forks source link

GPU tests are failing with latest Numba version #814

Closed aktech closed 1 year ago

aktech commented 2 years ago

It seems like the GPU tests have started to fail with the latest Numba version:

Ref: https://github.com/pystatgen/sgkit/runs/4906151881?check_suite_focus=true

self = <numba.cuda.compiler._KernelConfiguration object at 0x7fce286f1d10>
dispatcher = <numba.cuda.compiler.Dispatcher object at 0x7fce387b8e10>
griddim = (1, 1, 1), blockdim = (32, 32, 1), stream = 0, sharedmem = 0

    def __init__(self, dispatcher, griddim, blockdim, stream, sharedmem):
        self.dispatcher = dispatcher
        self.griddim = griddim
        self.blockdim = blockdim
        self.stream = stream
        self.sharedmem = sharedmem

        if config.CUDA_LOW_OCCUPANCY_WARNINGS:
            ctx = get_context()
            smcount = ctx.device.MULTIPROCESSOR_COUNT
            grid_size = griddim[0] * griddim[1] * griddim[2]
            if grid_size < 2 * smcount:
                msg = ("Grid size ({grid}) < 2 * SM count ({sm}) "
                       "will likely result in GPU under utilization due "
                       "to low occupancy.")
                msg = msg.format(grid=grid_size, sm=2 * smcount)
>               warn(NumbaPerformanceWarning(msg))
E               numba.core.errors.NumbaPerformanceWarning: Grid size (1) < 2 * SM count (80) will likely result in GPU under utilization due to low occupancy.

../../../../miniconda33/envs/test/lib/python3.7/site-packages/numba/cuda/compiler.py:724: NumbaPerformanceWarning
aktech commented 2 years ago

Since the tests are failing due to Numba warning, one quick fix is to disable low occupancy warnings (for tests):

from numba import config
config.CUDA_LOW_OCCUPANCY_WARNINGS = 0

Of course it would hide the issue, but at least it isolates the issue.