Open yimoj opened 9 months ago
manopapad
commented
9 months ago @lightsighter It looks like the issue is when we feed centers[d] == 0 and tile[d] == 0 (for some d) to the launch_small_tile_kernel function https://github.com/nv-legate/cunumeric/blob/branch-24.01/src/cunumeric/convolution/convolve.cu#L608 (which happens when one dimension of the filter array is 1). Any quick fix we could try?
lightsighter
commented
6 months ago I can't believe you guys are still using this code. 🙄
@yimoj try again with this patch:
diff --git a/src/cunumeric/convolution/convolve.cu b/src/cunumeric/convolution/convolve.cu
index 7d185d6d..33add3cc 100644
--- a/src/cunumeric/convolution/convolve.cu
+++ b/src/cunumeric/convolution/convolve.cu
@@ -803,7 +803,8 @@ __host__ void direct_convolution(AccessorWO<VAL, DIM> out,
for (int d = DIM - 1; d >= 0; d--) {
// Make sure that each tile is at least double the size of the filter
// so that we can get some savings in bandwidth needed
- tile[d] = 2 * centers[d];
+ tile[d] = 2 * extents[d];
+ assert(tile[d] > 0);
if (d == (DIM - 1)) {
// In order to maximize bandwidth, we want to make sure we're loading at
// least 128B of contiguous memory along the last axis (row-major) of input
@@ -1344,7 +1345,8 @@ __host__ static inline void cufft_convolution(AccessorWO<VAL, DIM> out,
for (int d = DIM - 1; d >= 0; d--) {
// Make sure that each tile is at least double the size of the filter
// so that we can get some savings in bandwidth needed
- tile[d] = 2 * centers[d];
+ tile[d] = 2 * extents[d];
+ assert(tile[d] > 0);
if (d == (DIM - 1)) {
// In order to maximize bandwidth, we want to make sure we're loading at
// least 128B of contiguous memory along the last axis (row-major) of inputI'm pretty sure it should always have been like this, was just a typo.
SIGABRT trace
I'm unable to reproduce this failure mode with the latest internal core and cunumeric.
Software versions
Python : 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:49:32) Platform : Linux-5.15.0-79-generic-x86_64-with-glibc2.31 Legion : v23.11.00.dev-29-g57265c0a Legate : 23.11.00.dev+29.g57265c0a Cunumeric : 23.11.00.dev+20.g6fda4437 Numpy : 1.26.0 Scipy : 1.11.3 Numba : 0.58.1 CTK package : cuda-version-12.0-hffde075_2 (conda-forge) GPU driver : 550.00 GPU devices : GPU 0: NVIDIA A2 GPU 1: NVIDIA A10 GPU 2: NVIDIA A10 GPU 3: NVIDIA A40
Jupyter notebook / Jupyter Lab version
No response
Expected behavior
For the two shapes below convovle and print works fine with NumPy/SciPy:
Observed behavior
SIGABRT
SIGFPE
Example code or instructions
And then the code above.
Stack traceback or browser console output
SIGABRT trace
SIGFPE trace