search

starcraft2-ai / rl-battle

GNU General Public License v3.0
2 stars 1 forks source link

Create better-formed benchmark table #22

Closed jimmyahacker closed 6 years ago

jimmyahacker commented 6 years ago

The current benchmark table is great, but we could do better. We might need to merge some cells to make it look cleaner like the DeepMind Table

Willian-Zhang commented 6 years ago

DeepMind Source Code

\begin{table}[]
~
\centering
\caption{Aggregated results for human baselines and agents on mini-games. All agents were trained for 600M steps.
\textsc{Mean} corresponds to the average agent performance, 
\textsc{best mean} is the average performance of the best agent across different hyper-parameters, \textsc{max} corresponds to the maximum observed individual episode score.
}
\label{tbl:minigames}
\begin{tabular}{c|c|rrrrrrr}
    \sc \textbf{Agent}
    & \sc \textbf{Metric} 
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{MoveToBeacon}}
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{CollectMineralShards}}
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{FindAndDefeatZerglings}}
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{DefeatRoaches}}
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{DefeatZerglingsAndBanelings}}
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{CollectMineralsAndGas}}
    & \multicolumn{1}{c}{\sc \rotatebox[origin=l]{90}{BuildMarines}}
    \\ \hline
\multirow{2}{*}{\sc Random policy}
    & \sc mean      & 1    & 17 & 4    & 1    & 23    & 12      & $<1$  \\
    & \sc max       & 6    & 35 & 19   & 46   & 118   & 750     & 5  \\ \hline
\multirow{2}{*}{\sc Random search}   
    & \sc mean & 25   & 32 & 21   & 51   & 55    & 2318    & 8   \\
    & \sc max       & 29   & 57 & 33   & 241  & 159   & 3940    & 46  \\ \hline
\multirow{2}{*}{\sc DeepMind human player}
    & \sc mean      & 26   & 133 & 46   & 41   & 729   & 6880    & 138  \\
    & \sc max       & 28   & 142 & 49   & 81   & 757   & 6952    & 142  \\ \hline
\multirow{2}{*}{\sc StarCraft GrandMaster}
    & \sc mean      & 28   & 177 & 61   & 215  & 727   & 7566    & 133   \\
    & \sc max       & 28   & 179 & 61   & 363  & 848   & 7566    & 133   \\ \hline\hline
\multirow{2}{*}{\sc Atari-net}

    & \sc best mean &  25   & 96 & 49   & 101  & 81    & 3356    & $<1$ \\
    & \sc max       & 33   & 131 & 59   & 351  & 352   & 3505    & 20  \\ \hline
\multirow{2}{*}{\sc FullyConv}
    & \sc best mean & 26   & 103 & 45   & 100  & 62    & 3978    & 3   \\
    & \sc max       & 45   & 134 & 56   & 355  & 251   & 4130    & 42  \\ \hline
\multirow{2}{*}{\sc FullyConv LSTM}
    & \sc best mean & 26   & 104 & 44   & 98   & 96    & 3351    & 6   \\
    & \sc max       & 35   & 137 & 57   & 373  & 444   & 3995    & 62
\end{tabular}
\end{table}
Willian-Zhang commented 6 years ago

should be moved to comparison repo

Willian-Zhang commented 6 years ago

moved to https://github.com/starcraft2-ai/comparison/issues/4