added if else statement to handle post_act_ln

kyegomez / BitNet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

https://discord.gg/qUtxnK2NMf

MIT License

1.69k stars 155 forks source link

Closed Hiromasa-H closed 6 months ago

Hiromasa-H commented 6 months ago

Description: To address an issue where instantiating BitFeedForward() with post_act_ln=False will result in TypeError: 'NoneType' object is not callable, I've added an if-else statement to create different sequences depending on the value of post_act_ln. Please see the issue for more details.
Issue: https://github.com/kyegomez/BitNet/issues/43, https://github.com/kyegomez/BitNet/issues/53
Dependencies: None

Hiromasa-H commented 6 months ago

I feel the solution presented in the original issue may be much cleaner. I will update the code if necessary.