Open danielzgtg opened 1 year ago
First of all Intel GPU has quite poor performance on itself. How many compute units the GPU has? Show clinfo output? It isn't even clear if GPU has more gflops than CPU.
run dlprim_flops
on this device to see general performance.
Also mnist is really tiny network that does benefit much of even high performance GPU. On the other hand pytorch comes with highly optimized CPU implementation from Intel. So it is hard to beat CPU in this case.
In any case, using channel first memory order pytorch is using the performance of my implementation is actually is good and sometimes even better than GPU routines implemented by Intel.
See this issue: https://github.com/oneapi-src/oneDNN/issues/1194 regarding the Intel's implementation of GPU operations performance.
First of all, thanks for the project! It's really cool that pytorch can utilize opencl now (I'm not fan of nvidia, and very unlikely will have their video card).
I have very similar setup and results - opencl on intel video card is more than 3 times slower than when executing on 4 cores of cpu. But I also have an older laptop which has a discrete amd card with mesa clover opencl. And there I have ~1.5 performance boost comparing to cpu (which is still slower than cpu on newer laptop). But probably it all will depend on a task that is executed.
How many compute units the GPU has? Show clinfo output? It isn't even clear if GPU has more gflops than CPU.
Here what I have for the intel card:
Max compute units 24
Max clock frequency 1150MHz
Device Partition (core)
Max number of sub-devices 0
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 256x256x256
Max work group size 256
Preferred work group size multiple (device) 32
Preferred work group size multiple (kernel) 32
Max sub-groups per work group 32
Sub-group sizes (Intel) 8, 16, 32
I also have a old desktop with an AMD card. It's a shame that it's stuck on OpenCL 1 with Clover, but Rusticl has OpenCL 3. Mesa 22.3 should be out by December.
OpenCL 1 with Clover,
AFAIR it is OpenCL 1.1. I tested rx560 with Clover driver and it worked fine - was somewhat slower than ROCm but yet worked.
Dear Artyom-beilis,
I am a huge fan of your fantastic work from the beginning. I am a volunteer to help tuning CLBlast for many devices (https://github.com/CNugteren/CLBlast/issues/1).
I know you also mentioned CLBlast before on Winograd or something related to it. If I remember your DLPrimitives kernel correctly, you are using your own GEMM. I am just wondering if it is properly tunned for Intel GPUs. I know Intel is good at tuning CPU gemm (MKL), however, their pervious opensource implementation on OpenCL with OpenCV were good for textbook. Therefore, to speed up its GPU, I guess CLBlast is good. After tuning with CLBlast, 3x performance boost were achieved with my iGPU (before tuning, GPU is worse than CPU). My testing was here: https://www.researchgate.net/figure/i7-1165g7-tuned-clblast_fig9_366394124. So, may I suggest adapting CLBlast to DLPrimitives to see if it gets more speedup on fully connected layers/Conv on im2col. In addition, it has been used in llamma.cpp to achieve comparative performance as cuBLAS. Please forgive me if I am wrong. I am just very interested in your fantastic work.
Best wishes, Jinchuan
1st of all I'm not sure that clblast has much advantage over dlprimitives. For example GEMM m=n=k=512 both give ~130-140 GFlops.
For example my gemm: Intel(R) UHD Graphics 630 [0x3e9b] on Intel(R) OpenCL HD Graphics When this GPU top GLOPS is around 401.413
dlprimitives sgemm
GEMM
NN 0: 512, 512, 512 136.9 GFlops (34.09%) 1.6 GB/s (11.07%) limited by gflops 34.09%
NT 0: 512, 512, 512 104.4 GFlops (26.01%) 1.2 GB/s ( 8.44%) limited by gflops 26.01%
TN 0: 512, 512, 512 190.9 GFlops (47.56%) 2.2 GB/s (15.44%) limited by gflops 47.56%
TT 0: 512, 512, 512 162.0 GFlops (40.35%) 1.9 GB/s (13.10%) limited by gflops 40.35%
clblast sgemm
NN: 143.772 GFLOPS
TN: 154.547
NT: 129.747
TT: 143.407
So it not necessary clear cut that clblast has better performance. Also note that when running convolition the im2col is actually integrated to the gemm itself so switching to clblast isn't trivial at all.
On the same note I need to mention that Intel's own deep learning library does not run well for channel first convolution since they just don't optimise for one. https://github.com/oneapi-src/oneDNN/issues/1194 Actually dlprimitives performs better on channel 1st order.
Hi, the problem is the size, for example MINIST has 60000 training samples, each with a size of 28x28 pixels. Hence, I am very interested in how this (60000x742 x 742x742 as an example to imitate the input and the weights to the hidden layer) performs compared to 512*512. Based on my experience, GPU gemm normally performs close to CPU when the size is small.
Best wishes
GPU gemm normally performs close to CPU when the size is small.
Which GPU do you have?
Typical max Intel Skylake GPU has ~400GFPLOPS which is quite a few. But typical multi core cpu has much more flops
If I run torch.matmul on same computer where I have 400GFLOPs GPU on CPU I get ~500GFLOPs performance (for mat mul i.e. way-way more than I can get from GPU.
Intel GPUs unfortunately aren't as powerful as modern data CPUs - they are good for graphics - less so for computation.
So I'm not surprised that you have better performance for CPU rather than on GPU.
Dear Artyom, I have several Intel iGPU and Arc GPUs in my lab. Take i7-1165G7 as an example, what I got is here: i7-1165g7.zip I admit that the code I implemented did not consider zero-copy for iGPU. All blue dots on both graphs are for iGPU. This is the case Intel make better GPU than CPU cores (CPU used Openblas).
Best wishes, Jinchuan
I have several Intel iGPU and Arc GPUs in my lab.
Nice. Can you post benchmark numbers for Arc GPUs? i.e run ./dlprim_flops
I have not managed to test it or tune for it.
CPU cores (CPU used Openblas).
OpenBLAS is somewhat smaller. I tested with torch.matmul and it is damn fast. The 500GFPLOPs was with torch.matmul. OpenBLAS's sgemm gives only 250 GFLOPS
Dear Artyom, Please let me interest you with something more interesting as on an AMD 7900XTX. I freshly build a windows dlprim_flops and receive the following result: 7900XTX.txt
I will try to make a test on ARC 770 16G when I find a free machine to plug off the CUDA/AMD card and then plug in the ARC card.
For torch, I believe they can use MKL for Blas if it is installed via Conda, Otherwise, it should be OpenBlas.
Best wishes, Jinchuan
Dear Artyom, I am also supervising an undergraduate student to do a comparative study on the performance of different GPU random number generators. May I ask if there is any paper/material that is related to the random kernel? (https://github.com/artyom-beilis/dlprimitives/blob/master/src/kernels/random.cl) Thank you very much! Best wishes, Jinchuan
I am also supervising an undergraduate student to do a comparative study on the performance of different GPU random number generators. May I ask if there is any paper/material that is related to the random kernel?
It is fairly standard Philox-4x32, see here the paper: https://www.thesalmons.org/john/random123/papers/random123sc11.pdf
I mentioned it in API: https://github.com/artyom-beilis/dlprimitives/blob/master/include/dlprim/core/common.hpp#L47
Dear Artyom, Please let me interest you with something more interesting as on an AMD 7900XTX. I freshly build a windows dlprim_flops and receive the following result: 7900XTX.txt
Best wishes, Jinchuan
Cool. It looks like my measurement of FLOPS are way incorrect it should be much more capable GPU according to wikipedia. That is why I get more than 100% flops for some GEMM ops.
Apple M2 MAX.txt HDF5_cpp is not built by default according to its website, hence no such lib from homebrew. Thus, disabled HDF5 lib in the makefile to get there.
I am also supervising an undergraduate student to do a comparative study on the performance of different GPU random number generators. May I ask if there is any paper/material that is related to the random kernel?
It is fairly standard Philox-4x32, see here the paper: https://www.thesalmons.org/john/random123/papers/random123sc11.pdf
I mentioned it in API: https://github.com/artyom-beilis/dlprimitives/blob/master/include/dlprim/core/common.hpp#L47
Thank you very much!
Dear Artyom, https://github.com/artyom-beilis/dlprimitives/blob/46b9d17b76c40d05b323a1a0ea484d61ac5f17b2/src/core/pointwise.cpp#L77C24-L77C25 would this be quite expensive to handling \n to \\\n
using for loop, or is this necessary? I did not find similar lines to specifically handle this in CLBlast which also used R" for kernels. https://github.com/CNugteren/CLBlast/blob/master/src/utilities/compile.cpp#L24
Best wishes, Jinchuan
The pointwise/broadcast is a special kind of kernels that are shortcuts for common simple operation: activations, various reductions needed for normalization, loss functions etc.
Here you get the actual line of code and embed it into a kernel, like here: https://github.com/artyom-beilis/pytorch_dlprim/blob/master/src/pointwise_ops.cpp#L155
Now, I agree that this code can be optimised, but it is also important to remember that since the GPU and CPU executions are asynchronous, as long as you can push kernels to execution queue faster than GPU can run them you don't bottleneck your system.
I appreciate the smart way you organised the global kernel parameters with different lengths. For me, I would write independent kernels for the above operations as well as prescan and other stuff. May I infer that all the kernels in dlprimitive are fully asynchronous? Is this the reason why I see this dim is restricted to 8D in https://github.com/artyom-beilis/dlprimitives/blob/master/src/kernels/broadcast_dims.h#L52C13-L52C15
Best wishes,
I would write independent kernels for the above operations
It is a problem if you need to support different kinds of broadcasts/reduces, like A.shape = [1,3,1,10], B.shape = [3,2,1] and you calc A+B -> shape = [1,3,2,10] - this kind of stuff requires dynamic code generation, add reduction over different dimensions like mean(A,dim=(...))
you can't write single kernel for them all.
The independent kernels are mostly for some computationally intensive/non-standard tasks like pooling, convolutions etc. So if there are tasks that can simplify coding :-)
I'll do it.
Currently the major bottle neck is actually implementing as many functions as needed and being able to do it simple is major help.
Is this the reason why I see this dim is restricted to 8D
Currently tensor is limited to 8 as constant and so far it was enough.
And what is more important it allows to guys who aren't familiar with GPU coding write some operations.
The independent kernels are mostly for some computationally intensive/non-standard tasks like pooling, convolutions etc.
There is another way which uses a fixed-length kernel. All the broadcasting size info regarding dimensions needs to be stored in a host buffer and then uploaded to the device as global mem to the kernel. The kernel is in charge of using those info to index the correct location of Array A and B. As a result, this kernel becomes synchronous due to the fact a blocking memory write needs to be issued to the command queue. This is fine for scientific computing tasks, but for AI especially with multiple in-platform GPUs, I guess your way of asynchronous is more welcome.
Intel Arc 770.txt as promised. This one really took time for it can freeze while executing some cases.
Thanks. Impressive GPU.
Can you give clinfo or it?
I wonder if it is possible to enable Winograd on one. It depends on shared memory configuration.
clinfo770.txt
You are welcome.
Platform Name: Intel(R) OpenCL Graphics
In the meantime, there is a website https://compubench.com/device.jsp?benchmark=compu20d&os=Windows&api=cl&D=Intel%28R%29+Arc%28TM%29+A770+Graphics&testgroup=info which could provide more results for new devices.
So I tried running this on the provided
mnist.py
. My main use case is that I'm on a laptop and I want it to train faster and use less battery. What I did not expect is that it was 3x slower using the GPU than using the CPU. I am onIntel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz
withintel-opencl-icd
. Furthermore, the GPU version caused the laptop fans to start spinning, but the CPU version didn't.Expected behaviour
Actual behaviour