Reverting a base change for Softmax introduced in a Jan 28 commit, that was missed during review.
Thanks Karthik for finding this! : )
Cleaned up the Polymax Section
a. removed PolymaxQuan -- should adhere to the ConSmaxQuan format
b. moved existing polymax to vpolymax -- keeping around since it is interesting that the shape worked
c. optimized the polymax version to minimize calculations per forward pass
Added boolean option to each relevant variation for dividing by the sequence length.
We just saw this paper two days ago, and seems there are some overlaps and hoping to see if similar strategy could work for language modeling targeted transformers:
Change Summary:
Thanks Karthik for finding this! : )
a. removed PolymaxQuan -- should adhere to the ConSmaxQuan format b. moved existing polymax to vpolymax -- keeping around since it is interesting that the shape worked c. optimized the polymax version to minimize calculations per forward pass
We just saw this paper two days ago, and seems there are some overlaps and hoping to see if similar strategy could work for language modeling targeted transformers:
https://arxiv.org/abs/2309.08586