Thoughts on GPU vs CPU

[wweevv-johndpope]

I'm assuming this can't be optimised to run on GPU (unless Postgres has some extension) How long does it take comparatively. Is it 100x slower?

[quassnoi]

I'm assuming this can't be optimised to run on GPU (unless Postgres has some extension)

No, it can't.

To make efficient use of the GPU, you would need to load the model parameters into the GPU memory and arrange them in a special way, as continuous arrays. Then you would run SIMT commands on the GPU.

They are something like "take these two continuous arrays of dwords, multiply together the first elements, the second elements, etc", and they are executed in parallel by multiple threads within the GPU. It only works if all these pairs are arranged just so in the memory.

PostgreSQL cannot work with GPU memory and cannot emit GPU commands. I can probably imagine a custom storage engine and a C function that would do that, but they would be of very little practical use.

How long does it take comparatively. Is it 100x slower?

Compared to the CPU on my 5 year old Dell, is about 2000x as slow. I didn't run it on a GPU.

[wweevv-johndpope]

I did see this - gpu + db https://github.com/BlazingDB/blazingsql

GPT in 500 lines of SQL would be interesting to see if this could run chatgpt - fyi @quassnoi wmalpica