Closed TsXor closed 1 year ago
vincefn
commented
1 year ago Could you specify exactly what version of pyvkfft and VkFFT (git hash) you are using? Also what is the exact GPU reference?
This is likely a VkFFT issue, so should be reported against that project, so @dtolm can have a look.
TsXor
commented
1 year ago Could you specify exactly what version of pyvkfft and VkFFT (git hash) you are using? Also what is the exact GPU reference?
This is likely a VkFFT issue, so should be reported against that project, so @DTolm can have a look.
pyvkfft version: 2022.1.1 Tested my modified version on GTX 1050 Ti and R7 240, GTX 1050 Ti works fine, but R7 240 fails all R2C test with ndim=2. Wheel is built on GTX 1050 Ti only.
TsXor
commented
1 year ago I found that in the failed R2C tests, for some unknown reason, the shape changed:
======================================================================
ERROR: test_r2c (pyvkfft.test.test_fft.TestFFT) (backend='pyopencl', n=30, dims=2, ndim=2, axes=None, dtype=dtype('float32'), norm=0, use_lut=None, inplace=True, r2c=True, dct=False)
Run R2C tests
----------------------------------------------------------------------
Traceback (most recent call last):
File "E:\Python\Python39\lib\site-packages\pyvkfft\test\test_fft.py", line 204, in run_fft
res = test_accuracy(backend, sh, ndim, axes, dtype, inplace,
File "E:\Python\Python39\lib\site-packages\pyvkfft\accuracy.py", line 431, in test_accuracy
else:
File "E:\Python\Python39\lib\site-packages\pyvkfft\accuracy.py", line 147, in l2
return np.sqrt((abs(a - b) ** 2).sum() / (abs(a) ** 2).sum())
ValueError: operands could not be broadcast together with shapes (30,30) (60,14)I found that in the failed R2C tests, for some unknown reason, the shape changed:
The change of shape is normal for inplace R2C transforms, but only along the last axis. The other axis should not change.
I just checked the pip-installed current version of pyvkfft and there are no issues with 2D R2C in single precision.
You need to give more details about your GPU and the exact software versions you are using
vincefn
commented
1 year ago To be more specific, if you start from a real array of shape (55, 30):
TsXor
commented
1 year ago I found that in the failed R2C tests, for some unknown reason, the shape changed:
The change of shape is normal for inplace R2C transforms, but only along the last axis. The other axis should not change.
I just checked the pip-installed current version of pyvkfft and there are no issues with 2D R2C in single precision.
You need to give more details about your GPU and the exact software versions you are using
GTX 1050 Ti and R7 240 on windows 10, pyvkfft version: 2022.1.1, compiler:MSVC, vkFFT.h is modified to avoid error (both using the modified version), both using pyopencl backend, R7 240 fails while GTX 1050 Ti does not. I build binary wheel on GTX 1050Ti machine, and installed it on both machines.
I also tried it on the integrated UHD630 (STILL ALL THE SAME PRESET), and it works fine.
TsXor
commented
1 year ago I found that in the failed R2C tests, for some unknown reason, the shape changed:
The change of shape is normal for inplace R2C transforms, but only along the last axis. The other axis should not change.
I just checked the pip-installed current version of pyvkfft and there are no issues with 2D R2C in single precision.
You need to give more details about your GPU and the exact software versions you are using
Also, I found that in the latest version of https://raw.githubusercontent.com/DTolm/VkFFT/master/vkFFT/vkFFT.h, the line
#pragma OPENCL EXTENSION cl_khr_int64 : enable\n\which caused errors has been deleted.
vincefn
commented
1 year ago Thanks for the details. Interesting that the same wheel fails on R7 240 and works onGTX 1050. I could understand a simple failure as the compute capabilities can be different, but I am puzzled by the error you are reporting on the size of the arrays. But maybe that's just a secondary error after another error.
@Dtolm do you have any idea what could go wrong ? Should the R7 240 work or are there know limitations ?
@TsXor you may want to try compiling the wheel on the R7 machine - just in case there is a subtle difference in the on-the-fly opencl kernel compile environement.
DTolm
commented
1 year ago Hello,
I have no idea what can go wrong with R7. It is quite an old GPU so something can be related to drivers. @TsXor can you post the latest VkFFT version results (there both LUT and nonLUT precision for 808x808 system should be the same)? Also, can you try compiling the VkFFT main repo with Vulkan backend and send the output of ./Vulkan_FFT -vkfft 15 ?
TsXor
commented
1 year ago I found something else: I tried fft of reikna on both machine, GTX 1050 Ti is normal, but R7 240 throws warning while compiling CL code: reikna_warning.log and after searching around, it seems to be the problem of old AMD GPUs: https://github.com/xmrig/xmrig/issues/1554#issuecomment-586700387
DTolm
commented
1 year ago I tried fft of reikna on both machine, GTX 1050 Ti is normal, but R7 240 throws warning while compiling CL code:
I am pretty sure these warnings are related to pyopencl, not VkFFT.
vincefn
commented
1 year ago The warnings are just generic opencl compilation warnings, they could be harmless.
So it's indeed likely the issue is due to old GPU or driver (>8 years old), since we know it works on more recent GPUs. A bit strange that it works on most transforms but not 2D R2C though.
At least good that the default automated tests caught that !
TsXor
commented
1 year ago Hello,
I have no idea what can go wrong with R7. It is quite an old GPU so something can be related to drivers. @TsXor can you post the latest VkFFT version results (there both LUT and nonLUT precision for 808x808 system should be the same)? Also, can you try compiling the VkFFT main repo with Vulkan backend and send the output of ./Vulkan_FFT -vkfft 15 ?
TsXor
commented
1 year ago The warnings are just generic opencl compilation warnings, they could be harmless.
So it's indeed likely the issue is due to old GPU or driver (>8 years old), since we know it works on more recent GPUs. A bit strange that it works on most transforms but not 2D R2C though.
At least good that the default automated tests caught that !
In fact this issue originates from my L0-Smoothing project. I tried it on the R7, and found output was not as expected (huge difference from cpu cauculated result), then I ran tests and tried to find out where was wrong.
And the driver version is 21.09.03.08-210511a-368456C-RadeonSoftware, I think that is relatively new.
DTolm
commented
1 year ago Hello, I have no idea what can go wrong with R7. It is quite an old GPU so something can be related to drivers. @TsXor can you post the latest VkFFT version results (there both LUT and nonLUT precision for 808x808 system should be the same)? Also, can you try compiling the VkFFT main repo with Vulkan backend and send the output of ./Vulkan_FFT -vkfft 15 ?
In Vulkan, 2D R2C + C2R work on R7 without any issues, which is good news. Can you also try OpenCL backend, by setting VKFFT_BACKEND to 3 in CMakeLists?
TsXor
commented
1 year ago Hello, I have no idea what can go wrong with R7. It is quite an old GPU so something can be related to drivers. @TsXor can you post the latest VkFFT version results (there both LUT and nonLUT precision for 808x808 system should be the same)? Also, can you try compiling the VkFFT main repo with Vulkan backend and send the output of ./Vulkan_FFT -vkfft 15 ?
In Vulkan, 2D R2C + C2R work on R7 without any issues, which is good news. Can you also try OpenCL backend, by setting VKFFT_BACKEND to 3 in CMakeLists?
In fact I am not compiling it because I don't know how to... I just used the release... And thank god that there's OpenCL version in release.
DTolm
commented
1 year ago In fact I am not compiling it because I don't know how to... I just used the release... And thank god that there's OpenCL version in release.
There is an OpenCL release: VkFFT_OpenCL_release_v1.2.30.zip
TsXor
commented
1 year ago In fact I am not compiling it because I don't know how to... I just used the release... And thank god that there's OpenCL version in release.
There is an OpenCL release: VkFFT_OpenCL_release_v1.2.30.zip
... I am somehow distracted these days, and just now I found that the last log I posted was on OpenCL version...
and here is the log for vulkan version: R7_Vulkan_15.log
DTolm
commented
1 year ago and here is the log for vulkan version:
I am confused, haven't you posted Vulkan benchmark results before? The last benchmark you sent also worked in 2D except that it crashed on a big system that takes almost 2 GB of memory.
TsXor
commented
1 year ago and here is the log for vulkan version:
I am confused, haven't you posted Vulkan benchmark results before? The last benchmark you sent also worked in 2D except that it crashed on a big system that takes almost 2 GB of memory.
I found the last result I got was using OpenCL versionš¤£ I mean, r7_vk15.log is OpenCL log. ps: R7 240 has only 2GB vmem
DTolm
commented
1 year ago ps: R7 240 has only 2GB vmem
That's what I meant about the possible reason why it failed. But still, It passes tests that fail in pyvkfft OpenCL. Maybe it is related to some synchronization specifics in OpenCL that I am not familiar with.
TsXor
commented
1 year ago Something off-topic: I finally found the problem in my program: I passed a pyopencl boolean array to kernel function and wrote like this in the kernel function:
__global bool * arr_mask,This is how my bug got hardware-dependent: boolean of opencl have different length on different hardwares, and boolean array of pyopencl is stored as np.int8.
So I converted the array to np.int at python side and receive array as int in kernel function, and my bug got solved.
But it's my program, which have nothing to do with the test suite, so this is off-topic.
vincefn
commented
1 year ago I finally managed to reproduce the issue:
ValueError: operands could not be broadcast together with shapes (30,30) (60,14)
As I already mentioned, the tests run fine on macOS (M1 with opencl) and linux (all backends), but the opencl 2D R2C tests do fail on windows (on my GTX 1080, so it's not a GPU age issue). (not sure about cuda, my windows pycuda install is not working at the moment)
This is rather strange, as there is almost no platform-dependent code. And it's not a pyopencl version issue.
vincefn
commented
1 year ago Hmm, correction: this seems a little more mundane - the error occurs when scipy is not installed (regardless of the platform), so that numpy r2c transforms are used instead of scipy's.
vincefn
commented
1 year ago And the reason is indeed the different behaviour between numpy and scipy rfftn:
In [5]: from numpy.fft import rfftn, irfftn
In [6]: a=np.zeros((30,10))
In [7]: b=rfftn(a)
In [8]: a.shape,a.strides
Out[8]: ((30, 10), (80, 8))
In [9]: b.shape,b.strides
Out[9]: ((30, 6), (16, 480))
In [10]: from scipy.fft import rfftn, irfftn
In [11]: b=rfftn(a)
In [12]: b.shape,b.strides
Out[12]: ((30, 6), (96, 16))So after a (2D) scipy rfftn, the fast axis remains the last dimension, but for numpy, the fast axis changes... And using simply pyopencl.array.Array.view() will change the number of elements along the fast axis.
This leads to:
view() after an r2c transform changes the last dimension onlyAre there cases where we would want the last axis not to be the fast axis ? Right now pyvkfft assumes that the last axis is the fast axis.
My GPU supports
cl_khr_fp64, but does not supportcl_khr_int64, so I made a small modification invkFFT.h: