Question about documentation in probability_of_improvement

google-research / rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

https://agarwl.github.io/rliable

Apache License 2.0

765 stars 47 forks source link

Question about documentation in probability_of_improvement #10

Closed zhefan closed 2 years ago

zhefan commented 2 years ago

Hi, I wonder if the documentation in probability_of_improvement function in metrics.py is wrong? Specifically,

scores_x: A matrix of size (num_runs_x x num_tasks) where scores_x[m][n] represent the score on run n of task m for algorithm X. https://github.com/google-research/rliable/blob/cc5eff51cab488b34cfeb5c5e37eae7a6b4a92b2/rliable/metrics.py#L77)

Should scores_x[n][m] be the score on run n of task m for algorithm X?

Thanks.

agarwl commented 2 years ago

You are right -- I think it should be that score_x[n][m] is score on run n on task m around looking at the code here: https://github.com/google-research/rliable/blob/master/rliable/metrics.py#L92. Will fix.

zhefan commented 2 years ago

Thanks for the clarification.