As a user, I would like to be able to generate a forecast progression plot

[epag]

---

Author Name: Hank (Hank) Original Redmine Issue: 45682, https://vlab.noaa.gov/redmine/issues/45682 Original Date: 2018-01-19

---

A forecast progression plot merely displays forecast time series by T0 (i.e., T0 in the legend) in a time series plot with observations displayed. Alex has some examples for the IOEP, which I'll attach later when we start to work on this.

Most forecast systems, including CHPS/FEWS, can generate forecast progression plots. Such plots, when also displaying observations, will likely show all of the observations and forecasts, not just those limited to pairs, and at their original time scale, though they may include features to adjust that scale. In other words, each time series will be loaded independently of the others and then displayed.

For the WRES forecast progression plot, based on an email exchange with James, I propose that we include only paired data in an evaluation version of the forecast progression plot. Thus, the plot would reflect the time series of pairs handed off to WRES for the pooled windows and in the desired time scale of the WRES output (i.e., after aggregating).

Obviously, the exact design for the plot is up for debate and I'm certainly open to other ideas.

Scheduled this for SR1 and prioritiy normal, reflecting that this may be slipped if we don't have time to do it. I've added Alex and James as watchers for now.

Hank

---

Redmine related issue(s): 118363

---

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-03-24T23:36:34Z

---

James,

I actually think we have much (all?) of the structure in place, now, to pull this off. Specifically, we can now store the pairs as time series. All we would need to do is push those time series to the wres-vis, which can readily plot them. (We can also add scatter plots for pairs at the same time.)

The problem is that I don't have time to do this. However, being optimistic, I'll schedule it for release 1.1 and maybe I'll be able to work on it.

Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2018-03-25T01:35:25Z

---

Yes, this shouldn't be difficult. We'd simply need to add a consumer to the @ProductProcessor@. It's a (small) wiring task, I believe. But, yes, not next week.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-05-04T15:13:02Z

---

Pushing this to Release 1.2 as something small that could be useful to users. I just don't see how this can be done in 1.1.

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2018-05-04T15:17:18Z

---

Should be pretty simple now we have a means of building generic @TimeSeries@, pairs or otherwise. If pairs, it's just a case of piggy-backing off the portion of wres-io that builds time-series of pairs and pushing that directly to wres-vis for plotting, perhaps through an identity metric or something like that if that's an easier route. Probably don't need to do anything incrementally, as I assume this would be for small datasets.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-09-06T20:06:42Z

---

Related to conversation earlier today as well where I suggest the client will open an evaluation with a request, close an evaluation with another request. Between the open and close, information can be interrogated about the evaluation that happened and the inputs. This is my suggestion for how to handle different kinds of information besides pure evaluation end-results from the system. I don't think we necessarily need to generate the plot itself but the information required to generate a plot might be nice.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-09-27T19:19:35Z

---

If WRES is in charge of producing numeric data, and display or reports are secondary (or for callers to handle), is this ticket still a thing?

In other words: what numeric data output is missing that prevents a caller/user/client from producing a forecast progression plot?

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2018-09-27T19:51:06Z

---

Jesse wrote:

If WRES is in charge of producing numeric data, and display or reports are secondary (or for callers to handle), is this ticket still a thing?

In other words: what numeric data output is missing that prevents a caller/user/client from producing a forecast progression plot?

I cannot answer this with complete confidence until I see precisely the forecast progression plot that the user wants.

But, from the description provided, this type of plot uses pairs plus unpaired left observations (not paired left observations).

What is missing? As things stand, probably just the unpaired observations that produced the left side of the pairs. We supply pairs via a backdoor. We don't supply unpaired observations.

In future? It depends. I want to remove the pairs. I want a new verification product called "forecast progression plot", which would contain all the numbers required.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-09-27T19:57:23Z

---

I agree with James's future. I wonder if it would be reasonable that the "forecast progress plot" include the unpaired "left" values??? Hmmm...

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2018-09-27T20:03:14Z

---

Hank wrote:

I agree with James's future. I wonder if it would be reasonable that the "forecast progress plot" include the unpaired "left" values??? Hmmm...

Hank

Yes, I have no problem with that. The plot is fundamentally about evaluation, and the additional obs just aid interpretation (e.g. about the adequacy of the cadence or timestep of the forecasts).

I would be against adding an "observed time-series diagram", for example, because it has nothing to do with evaluation.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-09-27T20:26:52Z

---

Sounds good to me. Like it being urgent, since it may be useful in the context of ROE.

Hank

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-07T22:36:22Z

---

This sounds like a plot of pairs. Do you think we still should add this?

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2020-12-07T22:41:57Z

---

I think so. And a scatter plot too. These two things are probably the most rudimentary forms of evaluation that a user might start with (before any statistics) and we don't provide them. My only concern is the size of them. It makes the outputs bigger than the inputs, it adds data. But size is not an adequate reason to not do it, since a user can already request an evaluation of any amount of data.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2020-12-07T22:46:24Z

---

Then again, there really isn't much difference between these two things and "the pairs". Worth noting that our csv paired format doesn't preserve the time-series shape of the pairs, unlike the protobuf, but also that we don't message the pairs as protobufs yet, so we are stuck with csv pairs off-to-the-side. But the only difference between these plots and the pairs is creating a graphic from them and creating graphics is not core business. Hmmm.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2020-12-08T11:50:10Z

---

As author of the ticket, I feel like I should reply, though James largely answered for me. :)

Yes, we need to have a means by which our users can see a forecast progression plot or scatter plot of the pairs. How that happens, I don't know, but the requirement is there. Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2020-12-08T12:05:41Z

---

Hank wrote:

As author of the ticket, I feel like I should reply, though James largely answered for me. :)

Yes, we need to have a means by which our users can see a forecast progression plot or scatter plot of the pairs. How that happens, I don't know, but the requirement is there. Thanks,

Hank

If we were to add this to the graphics app, then it would require us to turn on the messaging of pairs first (which would, in turn, require making the pairs writer like any other message consumer, rather than something off to the side, plus it would need the new charting methods in the graphics app that consume the pairs). Regardless, the work to message the pairs needs a ticket - will add that.

In summary, how it happens:

1. Message the pairs; and
2. Add some new charting methods in the graphics app, one for each chart type.

[epag]

---

Original Redmine Comment Author Name: Chris (Chris) Original Date: 2020-12-08T14:03:11Z

---

Are we pretty much just talking about what the ISEDHydroGrapher spits out? If so... maybe we sit this one out. This sounds like feature bloat and it might be a better idea to have dedicated software for it rather than trying to bolt it on to an evaluation engine.

Would they be nice to have in the same place? No doubt; that adds a good bit of context.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2020-12-08T14:09:34Z

---

They're similar, I think, at least the progression plots (not the scatter plots). I guess the main difference is paired vs. unpaired and, more precisely, the same pairs that are used to compute statistics.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2020-12-08T14:12:41Z

---

Just saw James's response, but posting anyway since I spent the time to write it. :)

Chris wrote:

Are we pretty much just talking about what the ISEDHydroGrapher spits out?

Similar. I will say that scatter plot has been mentioned, as well, and that is not covered by ISEDHydroGrapher.

If so... maybe we sit this one out. This sounds like feature bloat and it might be a better idea to have dedicated software for it rather than trying to bolt it on to an evaluation engine.

Would they be nice to have in the same place? No doubt; that adds a good bit of context.

I agree that this is not necessarily a high priority, because other tools are filling in the gaps. To be clear, however, this would tie the forecast progression plot to an evaluation. It would provide the time series for only pairs contributing to calculating metrics and after rescaling the data to the @desiredTimeScale@ (correct me if wrong). Presumably it would/could also be related to the pools of pairs, though that may lead to some funky plots. The ISEDHydroGrapher is more of a forecast analytics tool, and is not tied to an evaluation.

Thanks,

Hank

[epag]

---

Original Redmine Comment Author Name: Chris (Chris) Original Date: 2020-12-08T15:04:39Z

---

I was just using the hydrographer as an example; I've seen three or four tools spitting out progression plots which had me wondering.

Regardless of what is planned for these sorts of plots, I agree that being able to pump out pair details as messages is a great start.

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-10-05T17:56:06Z

---

This is the development ticket for a forecast progression plot. It was discussed in the context of the challenges slide (interrogating raw data) from Seann presented during the 6th Users' Forum. James mentioned perhaps outputting it as HTML to allow for zooming, since it is likely to include a bunch of data spanning a long time period.

Hank

[epag]

---

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-10-05T20:13:51Z

---

James mentioned perhaps outputting it as HTML to allow for zooming, since it is likely to include a bunch of data spanning a long time period.

Chris saw this comment and pointed me to the ISEDHydroGrapher. It creates an HTML file that contains all of the data needed to visualize a progression plot using Plotly. I've attached an example (gzipped).

When I mentioned figuring out how to produce something like that in Java, he responded that it could be a Python microservice behind the COWRES. I'm picturing something similar to how the current events broker and graphics client works, except with the client being Python code that generates Plotly HTML that is portable and ready to be displayed. Not sure if it could use the same events broker, however: that is used for messaging statistics, not raw data, but the developers can figure that out.

Anyway, I think this proves the concept, and using a microservice architecture makes sense, its just a matter of identifying tooling.

Thanks,

Hank

P.S. ISEDHydroGrapher is in GitHub here:

https://github.com/NOAA-OWP/ISEDHydroGrapher

Here is some code that talks to the ISEDHydroGrapher running on ised-dev1 (domain omitted) and prepares the self contained HTML:

>>> import requests
>>> url = "http://nwcal-ised-dev1.[domain]:9773/nwm/hydrograph?lid=DRRC2&observations=on&thresholds=off&navigate=off&configuration=short_range&variable=streamflow&date_range=[PT10H,now]&colors=ed0909,0c5eed&format=text"
>>> response = requests.get(url)
>>> with open('./test_file.html', 'w') as test_file:
...     test_file.write(response.text)

If you use a browser to visit the endpoint above, you won't just be able to download the HTML. Chris had to use Python to get a full self-container HTML.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2023-10-05T20:28:15Z

---

Makes sense to me. Yes, it could (should) using the same messaging api. There is a placeholder for messaging the pairs (a separate ticket) and we can message the raw data too, no problem. Good excuse for developing the first python client - after that, we'll probably want to move the netcdf writing to python.