Add Chapter 0, Autograd section backprop test

alex-garvey commented 1 year ago

Going through the course now and just finished the backprop function. I realized that there were no tests to ensure that gradients were added if multiple nodes have the same parent.

To illustrate, the following code is in the solution:

        if parent not in grads:

            grads[parent] = in_grad

        else:

            grads[parent] += in_grad

If you were to replace that code with grads[parent] = in_grad

all of the current tests pass. The new test will throw an error, because it has a graph where b is the parent of both d and e.

This comes from going through the solution and not understanding at first why that addition was there! The course is great so far, thank you!

I believe a similar change is needed in the SERI-MATS-2023-Streamlit-pages repo, I'm happy to make a pull request there as well if this is approved.

callummcdougall commented 1 year ago

Thanks - that's a really good point (and I can imagine a lot of people missing that in their implementation). Very happy to add those tests

callummcdougall commented 1 year ago

Just to check, what do you mean about the SERI MATS repo? (this isn't associated with ARENA; I'm using it for the current project I'm working on w/ Neel Nanda & Arthur Conmy)

alex-garvey commented 1 year ago

I was referring to this, was not sure if needed a similar change. Fine either way. https://github.com/callummcdougall/SERI-MATS-2023-Streamlit-pages/blob/45c9ebef2cd9edaf0999ef2a7c1e179f46b178c4/transformer_lens/rs/arthur/interview/arena_backprop.py#L1722

callummcdougall commented 1 year ago

Ah I see - I think that was just Arthur doing some interview prep, not relevant to the public material.

callummcdougall / ARENA_2.0

Add Chapter 0, Autograd section backprop test #18