neulab / explainaboard_web

MIT License
8 stars 2 forks source link

add trend property for benchmark plots #554

Closed qjiang002 closed 1 year ago

qjiang002 commented 1 year ago

Benchmark plot: show datapoints for every date

Add a new property trend in BenchmarkViewConfig. trend='all' : show the score of every unique date trend='increase' : only show the scores and dates that are larger than the previous ones, so the plot is in ascending trend. This property will also be useful for configuring meta analysis (e.g. setting trend='all' to display metadata for every unique date).

For example, this is the views config of all the systems using sst2.

"views": [
        {
            "name": "Overall Mean",
            "operations": [
                {"op": "mean"}
            ],
            "trend": "all"
        },
        {
            "name": "Overall Mean Skip Group System",
            "operations": [
                {"op": "mean", "skip_group_system": true}
            ],
            "trend": "all"
        },
        {
            "name": "Mean by Dataset",
            "operations": [
                {"op": "mean", "group_by": ["dataset_name"]}
            ],
            "trend": "all"
        },
        {
            "name": "Mean by Dataset Skip Group System",
            "operations": [
                {"op": "mean", "group_by": ["dataset_name"], "skip_group_system": true}
            ],
            "trend": "all"
        }
    ]

The result benchmark plots are:

Screen Shot 2022-12-05 at 5 46 48 PM ![Screen Shot 2022-12-05 at 5 47 04 PM](https://user-images.githubusercontent.com/71625258/205762321-549796c4-f484-4 Screen Shot 2022-12-05 at 5 47 12 PM 662-ac64-7d365af31ec4.png) Screen Shot 2022-12-05 at 5 47 19 PM Screen Shot 2022-12-05 at 5 47 39 PM

qjiang002 commented 1 year ago

Looks good.

At some point we'll probably want to make the concept of "plotting" more general to not just cover date-by-date plots, probably in a future PR.

Yes I agree. I'll file this in an issue and go ahead to merge this.