mortvest / hastl

HaSTL: A fast GPU implementation of STL decomposition with missing values and support for both CUDA and OpenCL
MIT License
11 stars 5 forks source link

BUG:? clEnqueueNDRangeKernel failed with error code -5 (Out of resources) #4

Closed feefladder closed 1 year ago

feefladder commented 1 year ago

When trying to run bulk_csv.py, I got the opaque error "An internal error occured while running the GPU program". Setting STL to 'debug' gave the following additional info:

initializing the environment
Initializing the device
processing train_decomp_daily.csv
running stl
Running the program
Traceback (most recent call last):
  File "/home/feefladder/miniconda3/envs/hastl/lib/python3.11/site-packages/hastl/stl.py", line 205, in fit
    s_data, t_data, r_data = self._fut_obj.main(Y,
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/feefladder/miniconda3/envs/hastl/lib/python3.11/site-packages/futhark_ffi/__init__.py", line 158, in wrapper
    self._errorcheck(err)
  File "/home/feefladder/miniconda3/envs/hastl/lib/python3.11/site-packages/futhark_ffi/__init__.py", line 108, in _errorcheck
    raise ValueError(self._get_string(self.lib.futhark_context_get_error(self.ctx)))
ValueError: build/temp.linux-x86_64-cpython-311/hastl._stl_opencl.c:38787: OpenCL call
  clEnqueueNDRangeKernel(ctx->opencl.queue, ctx->mainzisegmap_intragroup_217594, 1, ((void *)0), global_work_sizze_237177, local_work_sizze_237181, 0, ((void *)0), ctx->profiling_paused || !ctx->profiling ? ((void *)0) : opencl_get_event(&ctx->opencl, &ctx->mainzisegmap_intragroup_217594_runs, &ctx->mainzisegmap_intragroup_217594_total_runtime))
failed with error code -5 (Out of resources)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/feefladder/c/Git/hastl/examples/bulk_csv.py", line 87, in <module>
    process_file(stl_obj, dataset, "{}_decomp_{}.csv".format(out_name, periodicity), n_p)
  File "/media/feefladder/c/Git/hastl/examples/bulk_csv.py", line 24, in process_file
    seasonal, trend, remainder = stl_obj.fit(data.T, n_p = n_p, q_s=999)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/feefladder/miniconda3/envs/hastl/lib/python3.11/site-packages/hastl/stl.py", line 230, in fit
    raise ValueError("An internal error occurred while running the GPU program") from from_err
ValueError: An internal error occurred while running the GPU program
Unreferencing block arr->mem (allocated as arr->mem) in space 'device': 0 references remaining.
39844928 bytes freed (now allocated: 481310880 bytes)

I've found the following SO questions that I think are related: https://stackoverflow.com/q/50953665 https://stackoverflow.com/a/40221197

EDIT: I forgot to mention that harmonic.py and loess_test.py both work when I set the backend to 'opencl'

CLinfo:

``` Number of platforms 1 Platform Name NVIDIA CUDA Platform Vendor NVIDIA Corporation Platform Version OpenCL 3.0 CUDA 12.2.138 Platform Profile FULL_PROFILE Platform Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_khr_gl_event cl_nv_create_buffer cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_nv_kernel_attribute cl_khr_device_uuid cl_khr_pci_bus_info cl_khr_external_semaphore cl_khr_external_memory cl_khr_external_semaphore_opaque_fd cl_khr_external_memory_opaque_fd Platform Extensions with Version cl_khr_global_int32_base_atomics 0x400000 (1.0.0) cl_khr_global_int32_extended_atomics 0x400000 (1.0.0) cl_khr_local_int32_base_atomics 0x400000 (1.0.0) cl_khr_local_int32_extended_atomics 0x400000 (1.0.0) cl_khr_fp64 0x400000 (1.0.0) cl_khr_3d_image_writes 0x400000 (1.0.0) cl_khr_byte_addressable_store 0x400000 (1.0.0) cl_khr_icd 0x400000 (1.0.0) cl_khr_gl_sharing 0x400000 (1.0.0) cl_nv_compiler_options 0x400000 (1.0.0) cl_nv_device_attribute_query 0x400000 (1.0.0) cl_nv_pragma_unroll 0x400000 (1.0.0) cl_nv_copy_opts 0x400000 (1.0.0) cl_khr_gl_event 0x400000 (1.0.0) cl_nv_create_buffer 0x400000 (1.0.0) cl_khr_int64_base_atomics 0x400000 (1.0.0) cl_khr_int64_extended_atomics 0x400000 (1.0.0) cl_nv_kernel_attribute 0x400000 (1.0.0) cl_khr_device_uuid 0x400000 (1.0.0) cl_khr_pci_bus_info 0x400000 (1.0.0) cl_khr_external_semaphore 0x9000 (0.9.0) cl_khr_external_memory 0x9000 (0.9.0) cl_khr_external_semaphore_opaque_fd 0x9000 (0.9.0) cl_khr_external_memory_opaque_fd 0x9000 (0.9.0) Platform Numeric Version 0xc00000 (3.0.0) Platform Extensions function suffix NV Platform Host timer resolution 0ns Platform Name NVIDIA CUDA Number of devices 1 Device Name Quadro M1000M Device Vendor NVIDIA Corporation Device Vendor ID 0x10de Device Version OpenCL 3.0 CUDA Device UUID c9d4ef51-2577-fa14-ef79-90b937af869d Driver UUID c9d4ef51-2577-fa14-ef79-90b937af869d Valid Device LUID No Device LUID 6d69-637300000000 Device Node Mask 0 Device Numeric Version 0xc00000 (3.0.0) Driver Version 535.104.05 Device OpenCL C Version OpenCL C 1.2 Device OpenCL C all versions OpenCL C 0x400000 (1.0.0) OpenCL C 0x401000 (1.1.0) OpenCL C 0x402000 (1.2.0) OpenCL C 0xc00000 (3.0.0) Device OpenCL C features __opencl_c_fp64 0xc00000 (3.0.0) __opencl_c_images 0xc00000 (3.0.0) __opencl_c_int64 0xc00000 (3.0.0) __opencl_c_3d_image_writes 0xc00000 (3.0.0) Latest comfornace test passed v2022-10-05-00 Device Type GPU Device Topology (NV) PCI-E, 0000:01:00.0 Device Profile FULL_PROFILE Device Available Yes Compiler Available Yes Linker Available Yes Max compute units 4 Max clock frequency 1071MHz Compute Capability (NV) 5.0 Device Partition (core) Max number of sub-devices 1 Supported partition types None Supported affinity domains (n/a) Max work item dimensions 3 Max work item sizes 1024x1024x64 Max work group size 1024 Preferred work group size multiple (device) 32 Preferred work group size multiple (kernel) 32 Warp size (NV) 32 Max sub-groups per work group 0 Preferred / native vector sizes char 1 / 1 short 1 / 1 int 1 / 1 long 1 / 1 half 0 / 0 (n/a) float 1 / 1 double 1 / 1 (cl_khr_fp64) Half-precision Floating-point support (n/a) Single-precision Floating-point support (core) Denormals Yes Infinity and NANs Yes Round to nearest Yes Round to zero Yes Round to infinity Yes IEEE754-2008 fused multiply-add Yes Support is emulated in software No Correctly-rounded divide and sqrt operations Yes Double-precision Floating-point support (cl_khr_fp64) Denormals Yes Infinity and NANs Yes Round to nearest Yes Round to zero Yes Round to infinity Yes IEEE754-2008 fused multiply-add Yes Support is emulated in software No Address bits 64, Little-Endian Global memory size 4238344192 (3.947GiB) Error Correction support No Max memory allocation 1059586048 (1010MiB) Unified memory for Host and Device No Integrated memory (NV) No Shared Virtual Memory (SVM) capabilities (core) Coarse-grained buffer sharing Yes Fine-grained buffer sharing No Fine-grained system sharing No Atomics No Minimum alignment for any data type 128 bytes Alignment of base address 4096 bits (512 bytes) Preferred alignment for atomics SVM 0 bytes Global 0 bytes Local 0 bytes Atomic memory capabilities relaxed, work-group scope Atomic fence capabilities relaxed, acquire/release, work-group scope Max size for global variable 0 Preferred total size of global vars 0 Global Memory cache type Read/Write Global Memory cache size 98304 (96KiB) Global Memory cache line size 128 bytes Image support Yes Max number of samplers per kernel 32 Max size for 1D images from buffer 134217728 pixels Max 1D or 2D image array size 2048 images Max 2D image size 16384x16384 pixels Max 3D image size 4096x4096x4096 pixels Max number of read image args 256 Max number of write image args 16 Max number of read/write image args 0 Pipe support No Max number of pipe args 0 Max active pipe reservations 0 Max pipe packet size 0 Local memory type Local Local memory size 49152 (48KiB) Registers per block (NV) 65536 Max number of constant args 9 Max constant buffer size 65536 (64KiB) Generic address space support No Max size of kernel argument 4352 (4.25KiB) Queue properties (on host) Out-of-order execution Yes Profiling Yes Device enqueue capabilities (n/a) Queue properties (on device) Out-of-order execution No Profiling No Preferred size 0 Max size 0 Max queues on device 0 Max events on device 0 Prefer user sync for interop No Profiling timer resolution 1000ns Execution capabilities Run OpenCL kernels Yes Run native kernels No Non-uniform work-groups No Work-group collective functions No Sub-group independent forward progress No Kernel execution timeout (NV) Yes Concurrent copy and kernel execution (NV) Yes Number of async copy engines 1 IL version (n/a) ILs with version printf() buffer size 1048576 (1024KiB) Built-in kernels (n/a) Built-in kernels with version Device Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_khr_gl_event cl_nv_create_buffer cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_nv_kernel_attribute cl_khr_device_uuid cl_khr_pci_bus_info cl_khr_external_semaphore cl_khr_external_memory cl_khr_external_semaphore_opaque_fd cl_khr_external_memory_opaque_fd Device Extensions with Version cl_khr_global_int32_base_atomics 0x400000 (1.0.0) cl_khr_global_int32_extended_atomics 0x400000 (1.0.0) cl_khr_local_int32_base_atomics 0x400000 (1.0.0) cl_khr_local_int32_extended_atomics 0x400000 (1.0.0) cl_khr_fp64 0x400000 (1.0.0) cl_khr_3d_image_writes 0x400000 (1.0.0) cl_khr_byte_addressable_store 0x400000 (1.0.0) cl_khr_icd 0x400000 (1.0.0) cl_khr_gl_sharing 0x400000 (1.0.0) cl_nv_compiler_options 0x400000 (1.0.0) cl_nv_device_attribute_query 0x400000 (1.0.0) cl_nv_pragma_unroll 0x400000 (1.0.0) cl_nv_copy_opts 0x400000 (1.0.0) cl_khr_gl_event 0x400000 (1.0.0) cl_nv_create_buffer 0x400000 (1.0.0) cl_khr_int64_base_atomics 0x400000 (1.0.0) cl_khr_int64_extended_atomics 0x400000 (1.0.0) cl_nv_kernel_attribute 0x400000 (1.0.0) cl_khr_device_uuid 0x400000 (1.0.0) cl_khr_pci_bus_info 0x400000 (1.0.0) cl_khr_external_semaphore 0x9000 (0.9.0) cl_khr_external_memory 0x9000 (0.9.0) cl_khr_external_semaphore_opaque_fd 0x9000 (0.9.0) cl_khr_external_memory_opaque_fd 0x9000 (0.9.0) NULL platform behavior clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV] clCreateContext(NULL, ...) [default] Success [NV] clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) No platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) Invalid device type for platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform ICD loader properties ICD loader Name OpenCL ICD Loader ICD loader Vendor OCL Icd free software ICD loader Version 2.2.14 ICD loader Profile OpenCL 3.0 ```

Any pointers would be very welcome, otherwise if I manage to fix this issue, I think other people might still appreciate this issue as a form of documentation :p

mortvest commented 1 year ago

You have most likely run out of registers on your GPU. hastl has been optimized for high-end discrete GPUs. You can try experimenting with parameters of STL() and fit().

feefladder commented 1 year ago

What would be a high-end GPU? I have so far tried on the following, and all gave this error:

mortvest commented 1 year ago

The implementation is optimized for maximal performance on double precision and has been tested on NVIDIA A100 and AMD MI100. I don't have access to a GPU right now and can't help you more, but you can try experimenting with the threshold values (jump_threshold_1, jump_threshold_2, q_threshold_1 and q_threshold_2) in order to make hastl choose different kernel versions.

feefladder commented 1 year ago

You won't believe it...

In stead of installing directly from PyPI, I installed from main like pip install https://github.com/mortvest/hastl/archive/main.zip and it worked even on the workstation (Nvidia P4000). I think I ran into "the issue with device memory" that was patched on 2 feb 2020, but didn't get updated on PyPI, since the latest version (0.1.7) is from jan 26...

I'll leave this open because the request would be to update PyPI :)

Thanks a lot for the quick replies!

mortvest commented 1 year ago

Thanks so much for the investigation, I'll make sure to update it when I find some time :smile:

mortvest commented 1 year ago

Should be fixed with v0.1.8