search

jafioti / luminal

Deep learning at the speed of light.
https://luminalai.com
Apache License 2.0
1.45k stars 90 forks source link

Autograd add grad error (failing test) #69

Open swfsql opened 3 months ago

swfsql commented 3 months ago

Related to #68

swfsql commented 3 months ago

Note: reviewing the fix code, I think the it's still not correct because ultimately every different index may have it's own index diff that precedent removals may have interfered with it. Currently, only the indexes above the "global minimal" get a index diff, but not all the would need to.

For example, if idx 10 is removed and then idx 3 is also removed, it would work OK. But if then idx 7 is removed, then only those above idx 7 should have a diff of -2, while then those like idx 5 (which is below 7) should have only a -1 diff. And that removal on idx 5 should also affect all diffs above idx 5, so then I think every index should have it's own idx diff.

swfsql commented 3 months ago

Another note to add is that although the status quo (main) code doesn't fail on the second test (test_add_grad_decreasing_idx_r2), I think it removes the wrong dimension. This would happen because the vector still has enough items, but the wrong item would get removed.