Make einsum! inplace

[GiggleLiu]

e.g. In order to differentiate over trace y = einsum("ii->", A), we can use the index exchange AD rule. We get the backward function for adjoints adjA = einsum("->ii", adjy).

This notation makes sense (it is broadcasting) except missing the shape information of output array. One regards indices that does not appear in input as broadcasting indices.

If we have einsum!("ii->", A, y), then the backward einsum!("->ii", adjy, adjA) is well defined. It is AD friendly to write inplace functions first.

How difficult it is to implement one in the branch exp-space? @under-Peter

[under-Peter]

Should be solved by e1903e1531672479a571484a48ec24635cca4346, although backwards probably doesn't work yet. I.e. having an empty input contraction.

[GiggleLiu]

Good, BTW, is that branch ready to merge? I find it is still in WIP status.

[under-Peter]

I want to add traces and partial-traces first. Probably just a pass before the existing einsumis called where all indices that are traces or partial traces are reduced - shouldn't be too hard (ingoring making it fast). Then operations like "->ii" should be added and exponential-space einsumshould have all the features we want and could be a reference for tests etc. Then we could merge to have a reference if that makes sense.

[GiggleLiu]

I wonder if iterating over all unique indices (including those in output tensor), cumulate products to output tensor (the definition of einsum) would be simpler to implement

Using Cartesian indices https://docs.julialang.org/en/v1/manual/arrays/index.html

Not sure the performance is good. At least the memory allocation is much less (Given output tensor, there should be no allocation at all).