notebooks: Add plots for hardened training run

tomtseng commented 1 year ago

Creates plots evaluating the first version of the dragonslayer (s349284096 from the hardened training run) and the second version (s497721856)
- The code for generating these specific plots were moved into functions that can be reused more easily for future experiments
Some non-hardened plots were regenerated too because we renamed checkpoint 127 from "Initial" to "cp127" in the paper
Also sets up code for generating the Elo plot for https://www.notion.so/chaiberkeley/match-cp-79-103-127-505-v1-16384-elo-eval-42060cb2a3734c8f90bbbbb08c5bd19f once that experiment finishes (we need to update our Elo evaluation in our paper to use Nora's config refactor (go_attack PR 61))

review-notebook-app[bot] commented 1 year ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

netlify[bot] commented 1 year ago

Deploy Preview for goattack canceled.

Name	Link
Latest commit	24074183a7e318139d683ca6fc57c92c09262f29
Latest deploy log	https://app.netlify.com/sites/goattack/deploys/637bdc242aac050008b8a5b2

tomtseng commented 1 year ago

Updated to include plots for adversary @ 497mil steps
Moved the plot-generating code I used into functions so that they can be more easily reused in the future

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:15Z ----------------------------------------------------------------

Line #8.            figsize=(2.64049, 2),

What's the significance of this number?

(The significance of 5.50107 used below is that it's the width of the page.)

_tomtseng commented on 2022-11-21T19:56:38Z_ ----------------------------------------------------------------

Not sure, this was copied from existing code. It's 48% of 5.501 so I guess it's just half the width of the page minus a small margin

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:16Z ----------------------------------------------------------------

Line #10.            gridspec_kw={"height_ratios": [10]},

height_rations with a list of length one doesn't do anything. This is my fault cause I used it in other charts, but it doesn't need to be here.

_tomtseng commented on 2022-11-21T20:02:26Z_ ----------------------------------------------------------------

thanks, will remove

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:17Z ----------------------------------------------------------------

Line #15.            * df.query(f"adv_name.str.contains('ov1') == False & victim_visits <= 128")

Should this be an f string?

_tomtseng commented on 2022-11-21T20:02:32Z_ ----------------------------------------------------------------

thanks, will remove

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:18Z ----------------------------------------------------------------

Line #25.    plt.ylabel("Adv-$\\texttt{cp127}$ win \\%")

This format seems weird. Why not a space between 'Adv' and 'cp127'?

_tomtseng commented on 2022-11-21T20:03:56Z_ ----------------------------------------------------------------

This is the plot for "what happens if we train an adversary solely on cp127 and then evaluate it against cp505?" I'll rename it the y-axis label to "cp127-trained adv. win %"

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:18Z ----------------------------------------------------------------

Line #33.            figsize=(2.64049, 2),

This number again

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:19Z ----------------------------------------------------------------

Line #8.    plot_adv_visit_sweep(df, df2, 64, "$\\texttt{Latest}$", "adv-vs-cp505-vary-visits2")

Should this plot be committed?

_tomtseng commented on 2022-11-21T20:06:56Z_ ----------------------------------------------------------------

yeah, this is the existing plot for Figure 8b and is already committed

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:20Z ----------------------------------------------------------------

Line #9.        gridspec_kw={"height_ratios": [10]},

Height ratios

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:21Z ----------------------------------------------------------------

Line #62.            labels[i] += " ($\\texttt{Original}$)"

This loop is very similar to one above. Could it be extracted to a function?

_tomtseng commented on 2022-11-21T20:11:14Z_ ----------------------------------------------------------------

There's definitely code duplication between this cell and the previous cell. I didn't spend time refactoring it because

We're less likely to rerun this Elo analysis than we are to rerun all the adversary evaluation plots
Once the experiment being processed by this cell is finished, assuming the results look reasonable we can probably just delete the previous cell and replace its results with this cell

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:22Z ----------------------------------------------------------------

Line #63.        plt.subplot(1, 1, 1)

This looks like it shouldn't be there

_tomtseng commented on 2022-11-21T20:12:38Z_ ----------------------------------------------------------------

will remove

review-notebook-app[bot] commented 1 year ago

View / edit / reply to this conversation on ReviewNB

UFO-101 commented on 2022-11-21T17:37:23Z ----------------------------------------------------------------

Line #79.                    method="beta",

Maybe add a comment linking to documentation explaining what this means:

https://www.statsmodels.org/dev/generated/statsmodels.stats.proportion.proportion_confint.html

_tomtseng commented on 2022-11-21T20:12:44Z_ ----------------------------------------------------------------

will do

UFO-101 commented 1 year ago

Thanks for these refactors. This code definitely looks much better.

tomtseng commented 1 year ago

Not sure, this was copied from existing code. It's 48% of 5.501 so I guess it's just half the width of the page minus a small margin

View entire conversation on ReviewNB

tomtseng commented 1 year ago

thanks, will remove

View entire conversation on ReviewNB

tomtseng commented 1 year ago

thanks, will remove

View entire conversation on ReviewNB

tomtseng commented 1 year ago

This is the plot for "what happens if we train an adversary solely on cp127 and then evaluate it against cp505?" I'll rename it the y-axis label to "cp127-trained adv. win %"

View entire conversation on ReviewNB

tomtseng commented 1 year ago

yeah, this is the existing plot for Figure 8b and is already committed

View entire conversation on ReviewNB

tomtseng commented 1 year ago

There's definitely code duplication between this cell and the previous cell. I didn't spend time refactoring it because

We're less likely to rerun this Elo analysis than we are to rerun all the adversary evaluation plots
Once the experiment being processed by this cell is finished, assuming the results look reasonable we can probably just delete the previous cell and replace its results with this cell

--- View entire conversation on ReviewNB

tomtseng commented 1 year ago

will remove

View entire conversation on ReviewNB

tomtseng commented 1 year ago

will do

View entire conversation on ReviewNB

AlignmentResearch / KataGoVisualizer