Accelerate Convolution with GPU.js

harshkhandeparkar commented 5 years ago

Kernel Convolution

This is the process of convolving a small matrix(called a kernel) of size 3*3 or 5*5 or any odd*odd size over a larger matrix(of pixels) to get values of pixels based on the adjacent pixels. This technique is used in Gaussian blur, mean blur, filters, edge detection and our general purpose convolution module.

Performance

This technique is very slow as it has to do a lot of processing of pixels and lots of math operations.

GPGPU

GPGPU stands for general purpose programming on GPU. This is a way to write functions which dispatch many parallel operations to the GPU to accelerate the process. It is mainly used for matrix multiplication, matrix transformation and convolution.

Kernel

A kernel, not to be confused with the convolved matrix, is a fancy word for a function run on the GPU.

GPU.js

GPU.js is a library which compiles simple js functions into a GPGPU language like GLSL and runs all the calculations in parallel on a GPU. If a GPU is not available, it will fallback to normal js on the CPU. This can be used not only for convolution but also for modules which manipulate many pixels at once. Maybe even for the pixelManipulation API.

Thank you!

Your help makes Public Lab better! We deeply appreciate your helping refine and improve this site.

To learn how to write really great issues, which increases the chances they'll be resolved, see:

https://publiclab.org/wiki/developers#Contributing+for+non-coders

harshkhandeparkar commented 5 years ago

@jywarren @publiclab/is-reviewers I want to work on this issue myself. I just want your approval.

harshkhandeparkar commented 5 years ago

Guys! I benchmarked gpu.js on my old pc. It showed me this

IMG_20190312_193127

GPU is 465 times faster than CPU !! A few notes

This was done on my old PC which is about 6 yrs old with an avg graphics card.(GT 630)
This was done in a controlled environment. I.e. latest version of firefox quantum, and no other process other than the browser was running and the browser had only 1 tab open.
The benchmark multiplies two 512*512 arrays which is huge!! It even made the graphics driver unresponsive mostly because it was an old pc and an old version of windows and an old GPU.
I haven't tested this on my new PC which has a budget graphics card(GT 1030) but still it 8 times better than this one.(geekbench4 showed 45k for the new pc graphics card and 8k for the old one)
Chrome showed less performance ~18 times. Don't know why.
This is crazy and can help accelerate edge detect and other modules. Since those modules are to be run on a new and powerful pc and only for technical purposes maybe non gpu users should not be allowed to use the edge detect module?

What are your thoughts? cc @jywarren @publiclab/is-reviewers

harshkhandeparkar commented 5 years ago

@tech4GT

harshkhandeparkar commented 5 years ago

Update! I benchmarked it on my new pc. It was 219 times faster than the cpu. This doesn't mean that its performance was worse than that of the old pc. The multiplier was small as the cpu of this pc is also better than that of the old pc. It was faster than the old pc.

jywarren commented 5 years ago

Hi Harsh, this is really great! Actually the step where we most need GPU is the webgl-distort module we're planning with @tech4gt. Would you be able to attempt implementing that module using GPU.js? That would be tremendously helpful!!!

On Tue, Mar 12, 2019 at 3:48 AM Harsh Khandeparkar notifications@github.com wrote:

Kernel Convolution

This is the process of convolving a small matrix(called a kernel) of size 33 or 55 or any odd*odd size over a larger matrix(of pixels) to get values of pixels based on the adjacent pixels. This technique is used in Gaussian blur, mean blur, filters, edge detection and our general purpose convolution module. Performance

This technique is very slow as it has to do a lot of processing of pixels and lots of math operations. GPGPU

GPGPU stands for general purpose programming on GPU. This is a way to write functions which dispatch many parallel operations to the GPU to accelerate the process. It is mainy used for matrix multiplication, matrix transformation and convolution. Kernel

A kernel, not to be confused with the convolved matrix, is a fancy word for a function run on the GPU. GPU.js

GPU.js https://github.com/gpujs/gpu.js is a library which compiles simple js functions into GPGPU and runs all the calculations in parallel on a GPU. If a GPU is not available, it will fallback to normal js on the CPU. This can be used not only for convolution but also for modules which manipulate many pixels at once. Maybe even for the pixelManipulation API.

Thank you!

Your help makes Public Lab better! We deeply appreciate your helping refine and improve this site.

To learn how to write really great issues, which increases the chances they'll be resolved, see:

https://publiclab.org/wiki/developers#Contributing+for+non-coders

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/849, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ_FLEiN2Lf5UYN2_yztbfUx7wsGzks5vV1uOgaJpZM4bqQfw .

harshkhandeparkar commented 5 years ago

GPU.js only runs js simple functions on the GPU. I don't know how an implementation of canvas can be made here. Btw it currently uses the WebGl api to run the GLSL shader files which it compiles the js functions into. Support for openCL in node.js will be added soon. Maybe we will see a new release in the next few weeks. This new release will have GPU acceleration available in node.js which is currently unavailable. This new implementation will have an option to either use the webgl api or openCL. Currently it falls back to using cpu in nodejs. Does webgl distort use webgl built-in functions of some sort or a lib? If it does then maybe implementing the lib in js will be difficult or compiling webgl shaders will be difficult. If webgl works with pure js code then, maybe it will be easier.

harshkhandeparkar commented 5 years ago

Gpu.js just compiles simple js code to simple GLSL shaders. That's it. Now they are thinking of using OpenCL by using some package called NOOOCL. I'm not sure what that package is or does but maybe that one can help in webgl-distort?

harshkhandeparkar commented 5 years ago

Ok i looked into the distort code. Fisheyegl is your library right? If anything then maybe fisheyegl itself can use gpu.js when openCL is implemented? And if they can implement opencl than maybe fisheyegl can use openCL instead of GLSL and run openCL in nodejs using NOOOCL and GLSL in browser using webgl api. How does that sound? If the webgl code is doable in js then maybe fisheyegl can use gpu.js to write the distort code in js and it will work both in browser and in nodejs but it will take time to compile js to openCL. If they are able to implement opencl, then fisheyegl code can become purejs and maybe the whole lib can be made a part of IS.

harshkhandeparkar commented 5 years ago

Ok i did some research. NOOOCL stands for nodejs object oriented openCL bindings. This is a nodejs package which runs openCL shaders on a gpu in nodejs. It is like the canvas/webgl api but it uses openCL instead of GLSL so it gets a webcl context instead of a webgl context. For this to work in the fisheyegl lib, openCL shaders for the same will have to be written and used in node and GLSL shaders can be used in the browser.

Link to the lib github https://github.com/unbornchikken/nooocl

jywarren commented 5 years ago

Actually webgl-distort is a different type of distortion - perspective, not lens distortion: https://github.com/jywarren/webgl-distort

Currently it is implemented using WebGL, so I would like to pursue a way to achieve that simply by supporting WebGL in node. However, if you're able to use GPU.js and a convolution approach (since it is a matrix transform, i believe?) and it works, I'm open to that solution too!!!

harshkhandeparkar commented 5 years ago

Oh cool! I looked into the code. I might be able to implement it. Btw gpu.js has a pre-release for openCL

jywarren commented 5 years ago

Awesome.

On Wed, Mar 13, 2019, 3:11 AM Harsh Khandeparkar notifications@github.com wrote:

Oh cool! I looked into the code. I might be able to implement it. Btw gpu.js has a pre-release for openCL

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/849#issuecomment-472305997, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ6kED7sOAxD1TelPyCzVKmEKc3A8ks5vWKSBgaJpZM4bqQfw .

harshkhandeparkar commented 5 years ago

Gpu.js v2 with node support is coming soon! They will be using headless-gl to use GLSL in node. Headless-gl is maintained well and the latest release was 4d ago. Gpu.js v2 can be expected in another week or two at max.

niravasher commented 5 years ago

This is really really amazing, this is a great thought @HarshKhandeparkar From where did you get the idea?

niravasher commented 5 years ago

I agree that using gpu.js enhances processing speed, so should we use it only to develop fisheyegl module or should make a function like changePixel or getNeighbouringPixel on which other modules can be built?

harshkhandeparkar commented 5 years ago

It actually is used for general purpose computing by default and not for webgl. To make fisheyegl work, we are going to use a package headless-gl and @Divy123 is working on this. I am going to accelerate the whole PixelManipulation API and convolution (in modules which use it).

harshkhandeparkar commented 5 years ago

From where did you get the idea?

I was just searching for gpu processing in js (mostly for ML purposes) to see how developed it was compared to other languages/frameworks and I found this.

harshkhandeparkar commented 5 years ago

I wanted everyone's thoughts on this and also wanted the devs to do a quick survey/test. I have created a codepen. Here is a link to its debug mode. Here is its code.

Please have a look at it, benchmark it on matrices of different sizes(it multiplies a matrix of size XbyX, rows=cols). Please let me know the benchmark results and also let me know if there were any errors or the browser crashed. Any size above 4096 can crash the browser. But this is an operation that takes exponentially more time as the matrix size increases but convolution is linear and it will take less time to process and will be less difficult to process. I may change it to also benchmark convolution.

@tech4GT I would love to get your input here. I also request you to have a look at the code and find bugs/vulnerabilities(if any) or suggest changes/features or ask questions.

cc @jywarren @publiclab/is-reviewers

harshkhandeparkar commented 5 years ago

cc of the above comment @niravasher @subhahu123 @VibhorCodecianGupta

harshkhandeparkar commented 5 years ago

The implementation of this in IS will also be similar. I want the views of others before I open a PR.

harshkhandeparkar commented 5 years ago

Updated links to the pen:

jywarren commented 5 years ago

Hi, @HarshKhandeparkar ! this looks cool... are you asking to brainstorm how we can catch and prevent crashes? Sometimes webgl has a "fallback" to rebuild the context if it fails, and pick up again where it left off...

harshkhandeparkar commented 5 years ago

I am not facing any errors. Just wanted to see if it crashes the browser which can happen with matrix sizes > 2048 in matrix multiplication.

It want to add a threshold let's say 4000by4000 after which convolution will not take place on a directly gpu but will happen in steps instead. I.e in batches of 1000by1000 each.

All numbers are arbitrary @jywarren

harshkhandeparkar commented 5 years ago

Currently I am facing an error with CPU fallback as it has some texture probs etc. It is a bit confusing and complicated as the same function will work differently on cpu and gpu.

Robert plummer (maintainer of gpu js) is helping me with this. I will fix that no problem.

The codepen will be half broken right now so just benchmark matrix multiplication but not convolution. Convolution benchmark is broken.

harshkhandeparkar commented 5 years ago

A maintainer of GPU.js once said to me

Convolution should run wickedly stupid fast using GPU.js

😂

This is 💯 % true. It could convolve a 4000x4000 array of float values with a 3x3 kernel in just 160 milliseconds 🤯🤯. It would take about 20-30 seconds for a cpu to do this.

It was benchmarked here

harshkhandeparkar commented 5 years ago

I am working on this! :rocket:

publiclab / image-sequencer