Closed blengerich closed 1 year ago
This is enabled for the contextualized.modules encoders, should we use that functionality or wait for a convenience parameter in easy modules? @blengerich
Where is it enabled? I'm hoping for an encoder called something like Linear
available in contextualized.modules.ENCODERS
.
I see, it's enabled using an MLP with layers=1 so we can easily add this to the Encoders.
It'd be great to have a convenience for linear encoders. This could be done by having a NGAM with depth 1 and width 1, or by creating a new encoder class with a flat beta.