right now we have:
position_encoding: bool = Field(
default=False,
description="Use a sin to mark relative words positions. "
"Necessary for non-RNN style models.",
)
position_encoding_type: PositionEncodingType = Field(
default=PositionEncodingType.SinusoidalInterleaved,
description="Type of positional encoding.",
)
And we use max_relative_positions with odd values (-1=rotary, -2=alibi) to cover new mechanisms.
We need to switch to
position_encoding_type = SinusoidalInterleaved, SinusoidalConcat, Rotary, Alibi
maybe add learnedpositionencoding, ....
vince62s
commented
1 week ago
right now we have: position_encoding: bool = Field( default=False, description="Use a sin to mark relative words positions. " "Necessary for non-RNN style models.", ) position_encoding_type: PositionEncodingType = Field( default=PositionEncodingType.SinusoidalInterleaved, description="Type of positional encoding.", )
And we use max_relative_positions with odd values (-1=rotary, -2=alibi) to cover new mechanisms.
We need to switch to position_encoding_type = SinusoidalInterleaved, SinusoidalConcat, Rotary, Alibi maybe add learnedpositionencoding, ....