cmu-delphi / exploration-tooling

tools for evaluating and exploring forecasters
Other
0 stars 0 forks source link

External forecasters from Hub don't display in some breakdowns #100

Open brookslogan opened 6 months ago

brookslogan commented 6 months ago

With make download + make dashboard with settings including

TAR_PROJECT=covid_hosp_explore
EXTERNAL_SCORES_PATH=legacy-exploration-scorecards.qs

External forecasters don't display when x var is forecaster, ahead, or geo_value (no facet vars), but do appear when x var is forecast_date or target_end_date. Additionally, the size of the points with x var equal to forecaster seem a bit surprising; amoebalike has only ~1/4 the aheads and what looks like fewer forecast dates as well, and yet by forecaster has more points? (May tie into another issue about needing to intersect forecaster availability; will file.)

Example with x var as forecaster.

2024-02-23-172930_1885x1026_scrot

Example bookmark:

http://127.0.0.1:3838/?_inputs_&apply_exclusions=0&selected_forecasters=%5B%22external_scores_dbe6c55b%22%2C%22score_unspecialized.amoebalike%22%2C%22external_scores_41d0b4ba%22%5D&baseline=%22external_scores_5d3975fc%22&x_var=%22forecaster%22&facet_vars=null&excluded_geo_values=%5B%22as%22%2C%22gu%22%2C%22vi%22%2C%22mp%22%5D&excluded_aheads=null&selected_forecast_date_range=%5B%222020-01-01%22%2C%222024-02-23%22%5D&selected_target_end_date_range=%5B%222020-01-01%22%2C%222024-02-23%22%5D&selected_metric=%22wis%22&scale_by_baseline=false&facets_share_scale=true&.clientValue-default-plotlyCrosstalkOpts=%7B%22on%22%3A%22plotly_click%22%2C%22persistent%22%3Afalse%2C%22dynamic%22%3Afalse%2C%22selectize%22%3Afalse%2C%22opacityDim%22%3A0.2%2C%22selected%22%3A%7B%22opacity%22%3A1%7D%2C%22debounce%22%3A0%2C%22color%22%3A%5B%5D%7D&plotly_afterplot-A=%22%5C%22main_plot%5C%22%22&plotly_hover-A=null&plotly_relayout-A=%22%7B%5C%22width%5C%22%3A1570%2C%5C%22height%5C%22%3A1260%7D%22&plotly_click-A=%22%5B%7B%5C%22curveNumber%5C%22%3A5%2C%5C%22pointNumber%5C%22%3A258%2C%5C%22x%5C%22%3A18840%2C%5C%22y%5C%22%3A29.502735868025002%7D%2C%7B%5C%22curveNumber%5C%22%3A4%2C%5C%22pointNumber%5C%22%3A25%2C%5C%22x%5C%22%3A18841%2C%5C%22y%5C%22%3A35.63215342768691%7D%2C%7B%5C%22curveNumber%5C%22%3A3%2C%5C%22pointNumber%5C%22%3A34%2C%5C%22x%5C%22%3A18841%2C%5C%22y%5C%22%3A36.56844212852365%7D%5D%22&plotly_doubleclick-A=%22%5C%22main_plot%5C%22%22

They do show up here: 2024-02-23-173941_1876x1087_scrot

dsweber2 commented 6 months ago

Probably has something to do with only having aheads 15-28... There's likely something wrong with the upload script (also the place to address #102 incidentally). The current version shouldn't be rerun without filtering for duplicates.