This change adds additional option for layer initialization, providing more options for the user. In my personal work, I have found this to perform better than orthogonal.
closes #51
[X] I have raised an issue to propose this change (required for new features and bug fixes)
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Added a third layer initialization option of kaiming he
Description
Add a third option for weight_init as 'kaiming he' (see https://pytorch.org/docs/stable/nn.init.html and https://arxiv.org/pdf/1803.08494).
Motivation and Context
This change adds additional option for layer initialization, providing more options for the user. In my personal work, I have found this to perform better than orthogonal.
closes #51
Types of changes
Checklist
make format
(required)make format
formats the code but none of the formats are related to my changes.make check-codestyle
andmake lint
(required)make pytest
andmake type
both pass. (required)make doc
(required)Note: You can run most of the checks using
make commit-checks
.Note: we are using a maximum length of 127 characters per line