Closed vince62s closed 1 week ago
address #17 Now we need to configure "position_encoding_type" which takes the following values:
class PositionEncodingType(str, Enum): SinusoidalInterleaved = "SinusoidalInterleaved" SinusoidalConcat = "SinusoidalConcat" Learned = "Learned" Relative = "Relative" Rotary = "Rotary" Alibi = "Alibi" or None
For the first 3 (absolute position encoding) it will be handled at the Embeddings level.
For the last 3 it is handled in the multi-head-attention
By default it is: SinusoidalInterleaved
If you want no encoding then use None (or null in the config.json)
address #17 Now we need to configure "position_encoding_type" which takes the following values:
class PositionEncodingType(str, Enum): SinusoidalInterleaved = "SinusoidalInterleaved" SinusoidalConcat = "SinusoidalConcat" Learned = "Learned" Relative = "Relative" Rotary = "Rotary" Alibi = "Alibi" or None
For the first 3 (absolute position encoding) it will be handled at the Embeddings level.
For the last 3 it is handled in the multi-head-attention
By default it is: SinusoidalInterleaved
If you want no encoding then use None (or null in the config.json)