Open rockerBOO opened 1 year ago
Trying to figure out the best way to break up the blocks. Also need to look into how to identify the block appropriately. There are tools out there with in/mid/out "layers" or something like in DAAM and block weight merging. But the keys do not have these relevant associations so I'm not sure of the connections there.
This representation may be appropriate and then some cleaning up of what values should be together. Attention weights (k, v, q) are tricky to try and group separately as they associate in a process. Maybe each block should just be evaluated together in most cases and maybe a very very verbose version breaking down each individual component (like to_k, to_v, to_q, ,and so on) averages.
Also not sure if its even a good idea to break this up into blocks. But maybe this will be good enough for some specific purpose.
Let me know what you think.