Initial MHA native implementation

rwth-i6 / i6_models

Collection of NN-Model parts

Mozilla Public License 2.0

1 stars 0 forks source link

Initial MHA native implementation #51

Closed sleepyeldrazi closed 2 months ago

sleepyeldrazi commented 4 months ago

Adding a native Multi-Head Attention implementation. Currently only supports Self-Attention.
Adding Conformer module that uses the native Multi-Head Attention.
Added a test file for the implementation.

sleepyeldrazi commented 2 months ago

Changed the name of i6_models/parts/mha.py to i6_models/parts/mhsa.py to reflect the change of the naming, however does this introduce potential confusion due to i6_models/parts/conformer/mhsa.py sharing the name?

Atticus1806 commented 2 months ago

does this introduce potential confusion due to i6_models/parts/conformer/mhsa.py sharing the name?

It might but I still think this is the right position for the code.