Closed gkielian closed 2 months ago
The wrapped nn.linear uses a different initialization, would it be possible to have this match the previous nn.Linear initialization values?
Collaborator
Sounds good, will look into whether we can forward the initialization values from model.py
I tested too, and seems actually like it should be fine, will make some direct edits and merge in.
Merged in the latest repo changes and created a new PR with suggested adjustments:
These combine the latest PRs towards adding ability to experiment with KAN as a replacement for linear layers and for the MLP, while retaining backwards compatibility with prior features.
A small change was needed for the current KAN implementation, requiring a clamp to prevent div by zero.
After implementing the above, inference with the sample.py file became much more stable.