Closed lkct closed 3 months ago
NOTE: some Layer
s can have children Module
s or Parameter
s which might or might not be initialized by themselves.
We need to double-check what exactly it resets in reset_parameters
and how reset is called in __init__
.
We need to design whether a layer inits/resets its direct parameters or all children. We also need to check how this interacts with Reparameterization
, which is also a Module
This will be solved by adding symbolic initializer: only the user who is aware of the parameterization is capable of specifying a suitable initialization.
With reparam class added, the params do not necessarily mean the same as before, and the way to initialize them may need to be changed.
Therefore we need to double-check what exists in the repo (e.g.
reset_parameters
) and refactor those if needed.