search

google-research / rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.
https://agarwl.github.io/rliable
Apache License 2.0
765 stars 47 forks source link

typos in README #1

Closed MishaLaskin closed 3 years ago

MishaLaskin commented 3 years ago

Amazing work! Already using it for a project. Found a couple of typos. In the below part of the README median should be aggregate_median and plot_aggregate_metrics should be plot_interval_estimates`.

algorithms = ['DQN (Nature)', 'DQN (Adam)', 'C51', 'REM', 'Rainbow',
              'IQN', 'M-IQN', 'DreamerV2']
# Load ALE scores as a dictionary mapping algorithms to their human normalized
# score matrices, each of which is of size `(num_runs x num_games)`.
atari_200m_normalized_score_dict = ...
aggregate_func = lambda x: np.array([
  metrics.median(x),
  metrics.aggregate_iqm(x),
  metrics.aggregate_mean(x),
  metrics.aggregate_optimality_gap(x)])
aggregate_scores, aggregate_score_cis = rly.get_interval_estimates(
  atari_200m_normalized_score_dict, aggregate_func, reps=50000)
fig, axes = plot_utils.plot_aggregate_metrics(
  aggregate_scores, aggregate_score_cis,
  metric_names=['Median', 'IQM', 'Mean', 'Optimality Gap'],
  algorithms=algorithms, xlabel='Human Normalized Score')
agarwl commented 3 years ago

Thanks, @MishaLaskin for pointing out the typos! Fixed them in 73244387271f88ac12f27cc83af02484e3bc499f. Also, it's awesome that you already are using the library :) Also, the colab at bit.ly/statistical_precipice_colab may also be useful regarding how to use the library.