notebook: Add notebook estimate V100 GPU-days for cyclic adversary

tomtseng commented 1 year ago

Add notebook that parses the cyclic adversary's victimplay logs to estimate that the cyclic adversary used 2,223 V100 GPU-days of training for victimplay (training GPU not included)

review-notebook-app[bot] commented 1 year ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

netlify[bot] commented 1 year ago

Deploy Preview for goattack canceled.

Name	Link
Latest commit	9fe9237f87c8e9d98df18556538bddf2d7b3a7bc
Latest deploy log	https://app.netlify.com/sites/goattack/deploys/63f13797dfeaa40008913c6a

AdamGleave commented 1 year ago

Remember to rename from iclr2022 before merging!

tomtseng commented 1 year ago

Rerequesting review because I made several large changes:

Previously I was only looking at the victimplay GPU usage. I've added the train worker's GPU usage (not counting the time where the train worker is idling and waiting for more rows). The new V100 GPU-days estimate for the cyclic adversary is 2,223 days
Refactored things into functions so that we can get the passing adversary's training compute (3.07 V100 GPU-days)
Added a plot of GPU-days vs. win rate (against cp505-v4096h) for the cyclic-adversary.

The last few points of the plot are:

            adv_win     gpu_days
adv_steps                       
434127104  0.000000   634.901149
445174528  0.000000   656.277141
455012352  0.000000   674.590325
464995584  0.120000   693.807642
474976512  0.145833   712.869404
484961536  0.400000   731.345387
494657792  0.645833   749.644650
504708864  0.750000   781.309203
515616512  0.833333   832.093124
525883904  0.875000   931.111608
535865088  0.940000  1206.192888
545065216  0.957224  2223.228956
545993728  0.960000  2883.143384

So with the s535mil adv, we're achieving 94% win rate against cp505h-v4096 with 8% of KataGo's compute

tomtseng commented 1 year ago

Added plots of GPU-days vs. adversary training steps now (to replace the plots of FLOPs vs. adversary training steps in our paper draft)

tomtseng commented 1 year ago

Will take some time to read through the code closer later today. But I was wondering if we could also add the checkpoint switching lines to the win-rate vs compute plot? I'm thinking the lines will spread out towards the end which would be a nice effect to show off clearly.

good suggestion, added

tomtseng commented 1 year ago

Oops, I underestimated the pass adversary GPU-days by a factor of 7 (20.4 V100 GPU-days vs. 3.06 V100 GPU-days) because I forgot that the pass-adversary was run with a single victimplay worker with 7 GPUs rather than 7 victimplay workers with 1 GPU.

Updating paper draft now

AdamGleave commented 1 year ago

@tomtseng are you still waiting on a review for this, or is it blocked for some other reason?

ed1d1a8d commented 1 year ago

Ah yeah I wanted to take a look at this but never got around to it cause higher priority things kept popping up. Will try to make some time for this this week.

AlignmentResearch / KataGoVisualizer