Slerp instead of lerp

[ljleb]

During interpolation, the embeddings norm becomes smaller than either of the interpolated embeddings. We should instead consider rotating the features embeddings along an arc of the 2D spiral passing by the origin and any 2 embeddings to be interpolated.

To illustrate this idea, here's a 2D slice of how interpolation is done as of right now:

and here's a 2D slice of how interpolation should probably be done:

TL;DR: we need to pass the interpolation ratio through trigonometric functions and then normalize the length of the resulting vector.

[ljleb]

I ran some tests and curving around the origin does not yield better results. Now trying to rotate around the negative prompt.

[ljleb]

Rotating around the empty cond definitely seems to give more coherent results than rotating around 0.

[ljleb]

I just learned this concept actually has a name: slerp. This needs more tests to find which of the following points for the center of the circle gives the best results:

• 0
• empty string conditioning
• negative embedding (in the direction of CFG)

using the empty string conditioning as the center of the circle seems to give strictly superior results to using 0, having a similar structure but more focused details.