MoE doesn't work with reversible layers

lucidrains / routing-transformer

Fully featured implementation of Routing Transformer

MIT License

282 stars 29 forks source link

MoE doesn't work with reversible layers #13

Closed tomweingarten closed 3 years ago

tomweingarten commented 3 years ago

When reversible layers are on, the call to MoE fails because it gets the unexpected keyword "_reversible". Happy to provide a stack trace if that's helpful.

lucidrains commented 3 years ago

@tomweingarten should be fixed in the latest!

tomweingarten commented 3 years ago

Looks great, thanks!