search

kraiskil / onnx2c

Open Neural Network Exchange to C compiler.
Other
184 stars 30 forks source link

support for 3d MatMul #44

Closed AUTOMATIC1111 closed 2 months ago

AUTOMATIC1111 commented 3 months ago

This makes it possible to use MatMul nodes with 3d tensors; currently the program only supports 2d.

3d MatMul is needed for Attention layers. Some implementations use einsum, some 3d matmul, but neither are supported at the moment.

I tested the added code on the model I'm working on, and it works well, faster than pytorch on a single CPU thread. I didn't do any other tests.

kraiskil commented 2 months ago

Thanks for your patch. Looks good at a glance, I'll have a closer look soonest. (mostly on what 3d matmul even means. Last I checked, matrix multiplication was defined only for 2d. But times change :) )

I tried enabling the ONNX test for matmul3d. Seems to pass now. Matmul4d (whatever that is) does not.

Also, your contribution is significant enough to be copyrightable - if you want, please add yourself to the list (file LICENSE.txt.

AUTOMATIC1111 commented 2 months ago

All right. Thanks. 3d MatMul means you just have many matrices and you independently multiply them between each other like you normally do with 2d MatMul. Same with 4d, I'm sure.

kraiskil commented 2 months ago

Thanks :)