Clariffication: Pretrained Leaderboard Model

[nGQD]

Hello, thanks for the awe-inspiring work!

I'd be very grateful if you can kindly clarify whether the pretrained leaderboard model refers to TF++(all towns) as depicted in Table 12. I really appreciate any help you can provide.

[Kait0]

Yes, the 90DS model in Table 12 is the same model as the 53 DS model in Table 17 and also the same model as the 69 DS score model in Table 13. The 62 DS and the 66 DS model in Table 17 are the same as the 70 DS and the 73 DS model in Table 13 respectively.

[nGQD]

Thanks for answering, this really clears up my confusion.

Additionally, can you please clarify if the pretrained leaderboard model provided is trained on all towns instead of only the training towns?

[Kait0]

Yes all leaderboard models were trained with all publicly available towns since the benchmark has their own secret test towns. These are the same model that are evaluated on longest6 benchmark where models are evaluated in the training towns. Its perhaps a little confusing that TF++ in the longest6 folder and TF++WP is in the leaderboard folder in the relased weights. Both models were evaluated on longest6 and the leaderboard. The non-ensemble models submitted to the leaderboard is always seed 0. (Ensemble means all 3 seeds).

[nGQD]

Thanks for the awesome explanation!