unrealwill
commented
3 months ago It has a lot of parameters, so interpretability is hindered.
The high number of parameters allows for more efficient search for the right structure without getting stuck in local minimas.
When you stack them like MLP, they also probably like MLP discover hierarchical features ranging from low level to high level depending on their height in the stack.
As it is just KAN with functions expressed in an other basis, Fourier should have the same representation power that KAN.
KAN have full learnability of the activation functions, so it's one less prior inductive bias (aka you don't have to choose which activation function you would use), and the learned activation may be meaningful functions although expressed with a lot of parameters.
One area of research one can explore is the potential of KAN, (or fourier KAN) for grokking, aka finding the exact right structure.
One can experiment by generating synthetic data of a specific form, and try to recover the structure with a KAN.
Once the structure is recovered, one can then try to reduce the number of parameters by regularization, to gain some interpretability.
kaneyxx
Hi, thanks for this implementation. I'm curious about the interpretability. Can this Fourier layer be integrated into the original KAN and its interpretation tools?