As a developer, I want to identify and implement a solution for evaluating any number of model scenarios in a single evaluation run

[epag]

---

Author Name: Hank (Hank) Original Redmine Issue: 119109, https://vlab.noaa.gov/redmine/issues/119109 Original Date: 2023-08-04

---

See #118360 for the requirement being addressed and #115608 for the use case with slide examples.

This ticket will touch on various aspects of the code, so I'm not sure of the category. Setting it as baselines for now.

This ticket can be resolved once we have designed a solution and implemented it. Leaving as normal priority and in the backlog pending prioritization of the NWM team requirements for use case #115608.

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-08-04T13:49:20Z

---

I'm guessing this means multiple @predicted@ entries, one per model evaluated. I'm not sure what the impact will be on @baseline@. Is there only one? Or are the different models each serving as a baseline for comparison with the other models?

I'm going to estimate 128 staff hours for implementation. My only justification for that is that it "feels" like a quick ticket to address than the mapping ticket, #119105, and I estimated 256 for that.

Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-08-22T12:51:48Z

---

Agree with this estimate, for a first cut.

However, you cannot have multiple @predicted@ or @baseline@ or, more generally, multiple instances of the same declaration at the same level in yaml. Something like this should be possible, though:

observed: 
  sources: data/DRRC2QINE.xml
  variable: QINE
predicted:
  - label: HEFS
    sources: data/drrc2ForecastsOneMonthHEFS/
    variable: SQIN
  - label: ESP
    sources: data/drrc2ForecastsOneMonthESP/
    variable: SQIN
  - label: MMEFS
    sources: data/drrc2ForecastsOneMonthMMEFS/
    variable: SQIN
</code>

This would address the simple case where there is one @observed@ and up to N @predicted@.

For the more complex case involving explicit pairings of @observed@/@predicted@/@baseline@, we'd probably need a new aggregator (edit: not part of the first cut):

observed: 
  sources: data/DRRC2QINE.xml
  variable: QINE
predicted:
  sources: data/drrc2ForecastsOneMonthHEFS/
  variable: SQIN
  label: HEFS
multivariable:
  - observed: 
      sources: data/DRRC2QINE.xml
      variable: QINE
    predicted:
      sources: data/drrc2ForecastsOneMonthESP/
      variable: SQIN
      label: ESP 
  - observed: 
      sources: data/DRRC2QINE.xml
      variable: QINE
    predicted:
      sources: data/drrc2ForecastsOneMonthMMEFS/
      variable: SQIN
      label: MMEFS
</code>

In other words, we extend the existing, simple case of one pairing with other pairings under @multivariable@.

This way, the simple thing remains simple (at least one @observed@ and @predicted@ per evaluation), but the more complex thing is possible, an extension.