Closed TRillionZxY closed 3 years ago
As README.md mentioned, head_count
is an additional parameter that the paper did not explore. Setting head_count=1
would reproduce the paper's settings. in_channels
depends on the number of channels for your input. The common setting for the rest of the parameters are key_channels=in_channels // 2, value_channels=in_channels
.
Thank you!:)
When the head count is larger than one how do you adjust key_channels and value_channels?
When the head count is larger than one how do you adjust key_channels and value_channels?
No, you do not need to. The module automatically divides key_channels
and value_channels
by head_count
and uses the resultant head_key_channels
and value_key_channels
for each head. You do need to ensure that key_channels
and value_channels
are divisible by head_count
.
Hi, it's a great job. I'm a rookie for DL.
The EfficientAttention have four parameters: in_channels, key_channels, head_count, value_channels. I know key_channels = d_k, value_channels = d_v. What are head_count means? What is a common setting for four parameters?