Ignore 0th position

[nix-apollo]

Ignore the 0th sequence position

Description

• Ignore the relevant pos for gram and mean calc
• Zero out the relevant gradients for M_dash
• Zero out the gradients in edge calculation

This isn't well tested or carefully handled with asserts, but I'm using it for experiments.

Related Issue

313

Motivation and Context

How Has This Been Tested?

I added a single test to make sure it runs. I'll run and examine outputs.

Does this PR introduce a breaking change?

No

[stefan-apollo]

Discussed the motivation with Nix and said

• This will affect our ablation scores unless we account for that
  • Nix: So we would need to not ablate our model at the 0th position. But nothing exciting happens there
  • Stefan: That's actually fine we can do this.

Also Nix made a good unrelated argument

• This is actually a sensible RIB direction. Both W_pos[0] and W_E["\n"] are kinda "new semantic context" indicators
• Even with ignore_0th_position this direction is gonna be large

So we tentatively concluded we probably do not want to ignore the 0th position anymore, but if someone wants to finish the PR and implement the changed ablations that's also fine. Both are sensible RIB choices.

[stefan-apollo]

We decided to not implement this feature