search

GistNoesis / FourierKAN

MIT License
690 stars 57 forks source link

Always killed #1

Open HelloWorldLTY opened 5 months ago

HelloWorldLTY commented 5 months ago

Hi, thanks for your great work. I wonder if it is possible to provide some guidance about memory trade-off setting. I intend to allcoate a KAN with 3072 as input dimension and 2000 as output dimension. I always get killed. Normal MLP does not meet such error. Thanks a lot

unrealwill commented 5 months ago

When written naively, this layer consume a lot of memory. If you use einsum as explained in code comment, it should use less memory. Fusing with KeOps lazy tensor, is an alternative left to the reader.

The optimal memory usage is zero extra memory beyond the parameters (coeff size = 2inputdimoutputdim*gridsize) , inputs and outputs, which can be obtained by manually fusing :

https://github.com/GistNoesis/FusedFourierKAN

The code is not yet optimized, and painful to install, so performance may not be as fast as some code that use more memory.

Some other project https://github.com/Blealtan/efficient-kan maybe more user friendly with respect to memory.

HelloWorldLTY commented 5 months ago

Thanks a lot. I will try other versions.