dtcenter / METviewer

Tool that creates plots using MET verification statistics output and the R statistical package
http://www.dtcenter.org/met/metviewer/
Apache License 2.0
14 stars 1 forks source link

Bugfix for METviewer Scorecard #553

Open michelleharrold opened 1 month ago

michelleharrold commented 1 month ago

Replace italics below with details for this issue.

Describe the Problem

@briannen and I are working on applying interactive scorecards to MET output from the DTC RRFS Agile Framework Team. We are able to generate correct scorecards from older MET output (v10.1.1), but with more current (v11.1.0) MET output the scorecard does not generate correct output. A scorecard is created, but all cells are 'Not statistically relevant.'

Expected Behavior

Using a similar (but not the same, since the underlying MET output is different) scorecard XML, I would expect a correct scorecard with statistically relevant data for all cells with both METv10.1.1 and METv11.1.0 data.

Environment

Describe your runtime environment: Testing is taking place on Dakota. The scorecard script we are pointing to is: /d3/projects/METViewer/src_scorecard/apps/METviewer/bin/mv_scorecard.sh

A test XML using METv10.1.1 data is here: /d3/projects/RRFS/metv/interactive/scorecard_rrfs.xml (output: /d3/projects/RRFS/metv/interactive/scorecard_rrfs_sfc.png)

A test XML using METv11.1.0 data is here: /d3/projects/RRFS/metv/interactive/briannen/scorecard_test.xml (log: /d3/projects/RRFS/metv/interactive/briannen/test.log) (output: /d3/projects/RRFS/metv/interactive/briannen/scorecard_hwt_2mT.png)

To Reproduce

Describe the steps to reproduce the behavior: On dakota: /d3/projects/METViewer/src_scorecard/apps/METviewer/bin/mv_scorecard.sh /d3/projects/RRFS/metv/interactive/briannen/scorecard_test.xml

Relevant Deadlines

Ideally, we would like a fix by mid-November, but if not, we would like to have this fixed by Jan. for the Annual AMS meeting.

Funding Source

FY23 NOAA Base (2792533) [NOTE: Subject to change to NOAA FY24!]

Define the Metadata

Assignee

Labels

Milestone and Projects

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

bikegeek commented 1 month ago

Log message from the scorecard using METv11.1 data indicates there is an error: 2024-09-17T16:26:16,221 ERROR [-1] [addJsonRow] e.u.m.s.GraphicalOutputManager: WARNING: one of the fields contains '-'. The scorecard could be invalid!!!!!

This is traced to the edu.ucar.metviewer.scorecard.GraphicalOutputManager.java addJsonRow method. The model name is checked for the presence of more than one '-' and if that condition is met, raises a warning and then replaces '-' with whitespace.

This has been verified by a database query for data from METv10.x (mv_rrfs_href_spring2022) and METv11.1 (mv_hwt2024):

MET10.x data:

MariaDB [mv_rrfs_href_spring2022]> select distinct(model) from stat_header; +------------+ | model | +------------+ | HREF | | HREF_mean | | HREF_mem01 | | HREF_mem02 | | HREF_mem03 | | HREF_mem04 | | HREF_mem05 | | HREF_mem06 | | HREF_mem07 | | HREF_mem08 | | HREF_mem09 | | HREF_mem10 | | HREF_prob | | RRFS | | RRFS_mean | | RRFS_mem01 | | RRFS_mem02 | | RRFS_mem03 | | RRFS_mem04 | | RRFS_mem05 | | RRFS_mem06 | | RRFS_mem07 | | RRFS_mem08 | | RRFS_mem09 | | RRFS_mem10 | | RRFS_prob | +------------+

MET 11.1 data:

MariaDB [mv_hwt2024]> select distinct model from stat_header; +---------------------+ | model | +---------------------+ | GSL-MPAS05_mem000 | | HRRR_mem000 | | NSSL-MPAS-HN_mem000 | | NSSL-MPAS-HT_mem000 | | NSSL-MPAS-RT_mem000 | +---------------------+

The METv11.1 has model names with more than one '-' present. This is most likely causing inadvertent "mangling" of the DIFF_SIG model name in METv11.1 data (and most likely affecting the generation of correct/expected statistical results):

Note the absence of the minus sign between the two models for the METv11.1 data: DIFF_SIG(Z2 HRRR_mem000 350000 TMP ME-Z2 NSSL-MPAS-HN_mem000 350000 TMP ME)

Note the '-' to indicate the minus sign in the METv10.x data (which is expected/correct):

METv10.x: DIFF_SIG(Z10 HREF_mem04 360000 WIND BCRMSE-Z10 RRFS_mem01 360000 WIND BCRMSE)

michelleharrold commented 1 month ago

@bikegeek -- wooot! Thanks for tracking that down so quickly. Going forward, would it be possible to have more than one '-' present, or should we change our model name? If it is possible to only have one '-' present, we might want to document that, so users are aware.

Thank you!

bikegeek commented 1 month ago

I think moving forward we should allow model names with more than one '-'.

On Fri, Oct 25, 2024 at 4:12 PM michelleharrold @.***> wrote:

@bikegeek https://github.com/bikegeek -- wooot! Thanks for tracking that down so quickly. Going forward, would it be possible to have more than one '-' present, or should we change our model name? If it is possible to only have one '-' present, we might want to document that, so users are aware.

Thank you!

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/METviewer/issues/553#issuecomment-2438950810, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHRD2ASSXHHSLWTW2XDZ5K64FAVCNFSM6AAAAABQO5R366VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZYHE2TAOBRGA . You are receiving this because you were mentioned.Message ID: @.***>

bikegeek commented 3 weeks ago

The Error raised from the GraphicalOutputManager might not be the cause of the error, it might just be the check that indicates something is wrong. I've set up an IDE on seneca to step through the code to see where the DIFF_STAT statistics are being computed.

michelleharrold commented 3 weeks ago

thanks @bikegeek -- if you need someone to help test, work through this, please let me or @briannen know!

michelleharrold commented 1 week ago

After reloading a database that changed the model names with multiple dashes to have underscores instead, I was able to make a scorecard successfully! So this error does seem to be directly related to multiple dashes.