RNN problem

[davidweichiang]

Section 6.2 deals with the problem, most visible in RNNs, where you want a linear transformation from an axis to itself, which currently requires renaming. There are four solutions proposed:

• matrix axes (§6.1.3)
• contracting two names (§6.2.1)
• stars (§6.2.2)
• numbered axes (§6.2.3). I really liked this but I think everyone agrees that it's not very self-explanatory

and a fifth solution would be:

• adopt a style that avoids reusing axis names; even the timesteps of an RNN would have distinct names

I think I'd like to cut this down to just one, and to make it part of the main document.

[davidweichiang]

Branch https://github.com/namedtensor/notation/tree/dual shows what some of these options look like for RNN, self-attention, and MVN.

I think the two-name contraction operator might have won. It's simple and it eliminates the most renamings (all but one). It just doesn't look very nice, but hopefully it wouldn't be used too much.

[davidweichiang]

Do any of these look better?

@srush @boazbk

[srush]

The first is definitely my favorite.

Some people like that section though :)

https://twitter.com/ocramz_yo/status/1339985255185477632

[boazbk]

I also prefer the first!

[davidweichiang]

First meaning matrix-shaped axes, or the two-name contraction?

[boazbk]

I mean that

Is a good choice for dot product where we match the ax1 axis of A with the ax2 axis of B

[srush]

(also what I meant)

[davidweichiang]

Ah, ok. And among the different solutions, do you like that two-name contraction the best too?

[srush]

Yes. Its not the most interesting, but I think it fits our goals.