Token Downsampling (https://arxiv.org/abs/2402.13573) is similar to Token Merging but with less quality loss and a massively simpler implementation.
Perhaps combining it with agent attention will improve results of AgentSD?
PixArt-Sigma uses a very similar method in their paper (where they call it KV-Compression)
The reference implementation is available at https://github.com/ethansmith2000/ImprovedTokenMerge
Token Downsampling (https://arxiv.org/abs/2402.13573) is similar to Token Merging but with less quality loss and a massively simpler implementation. Perhaps combining it with agent attention will improve results of AgentSD? PixArt-Sigma uses a very similar method in their paper (where they call it KV-Compression) The reference implementation is available at https://github.com/ethansmith2000/ImprovedTokenMerge