Open epag opened 3 weeks ago
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-03-24T23:36:34Z
James,
I actually think we have much (all?) of the structure in place, now, to pull this off. Specifically, we can now store the pairs as time series. All we would need to do is push those time series to the wres-vis, which can readily plot them. (We can also add scatter plots for pairs at the same time.)
The problem is that I don't have time to do this. However, being optimistic, I'll schedule it for release 1.1 and maybe I'll be able to work on it.
Thanks,
Hank
Original Redmine Comment Author Name: James (James) Original Date: 2018-03-25T01:35:25Z
Yes, this shouldn't be difficult. We'd simply need to add a consumer to the @ProductProcessor@. It's a (small) wiring task, I believe. But, yes, not next week.
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-05-04T15:13:02Z
Pushing this to Release 1.2 as something small that could be useful to users. I just don't see how this can be done in 1.1.
Hank
Original Redmine Comment Author Name: James (James) Original Date: 2018-05-04T15:17:18Z
Should be pretty simple now we have a means of building generic @TimeSeries
Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-09-06T20:06:42Z
Related to conversation earlier today as well where I suggest the client will open an evaluation with a request, close an evaluation with another request. Between the open and close, information can be interrogated about the evaluation that happened and the inputs. This is my suggestion for how to handle different kinds of information besides pure evaluation end-results from the system. I don't think we necessarily need to generate the plot itself but the information required to generate a plot might be nice.
Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-09-27T19:19:35Z
If WRES is in charge of producing numeric data, and display or reports are secondary (or for callers to handle), is this ticket still a thing?
In other words: what numeric data output is missing that prevents a caller/user/client from producing a forecast progression plot?
Original Redmine Comment Author Name: James (James) Original Date: 2018-09-27T19:51:06Z
Jesse wrote:
If WRES is in charge of producing numeric data, and display or reports are secondary (or for callers to handle), is this ticket still a thing?
In other words: what numeric data output is missing that prevents a caller/user/client from producing a forecast progression plot?
I cannot answer this with complete confidence until I see precisely the forecast progression plot that the user wants.
But, from the description provided, this type of plot uses pairs plus unpaired left observations (not paired left observations).
What is missing? As things stand, probably just the unpaired observations that produced the left side of the pairs. We supply pairs via a backdoor. We don't supply unpaired observations.
In future? It depends. I want to remove the pairs. I want a new verification product called "forecast progression plot", which would contain all the numbers required.
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-09-27T19:57:23Z
I agree with James's future. I wonder if it would be reasonable that the "forecast progress plot" include the unpaired "left" values??? Hmmm...
Hank
Original Redmine Comment Author Name: James (James) Original Date: 2018-09-27T20:03:14Z
Hank wrote:
I agree with James's future. I wonder if it would be reasonable that the "forecast progress plot" include the unpaired "left" values??? Hmmm...
Hank
Yes, I have no problem with that. The plot is fundamentally about evaluation, and the additional obs just aid interpretation (e.g. about the adequacy of the cadence or timestep of the forecasts).
I would be against adding an "observed time-series diagram", for example, because it has nothing to do with evaluation.
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-09-27T20:26:52Z
Sounds good to me. Like it being urgent, since it may be useful in the context of ROE.
Hank
Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-07T22:36:22Z
This sounds like a plot of pairs. Do you think we still should add this?
Original Redmine Comment Author Name: James (James) Original Date: 2020-12-07T22:41:57Z
I think so. And a scatter plot too. These two things are probably the most rudimentary forms of evaluation that a user might start with (before any statistics) and we don't provide them. My only concern is the size of them. It makes the outputs bigger than the inputs, it adds data. But size is not an adequate reason to not do it, since a user can already request an evaluation of any amount of data.
Original Redmine Comment Author Name: James (James) Original Date: 2020-12-07T22:46:24Z
Then again, there really isn't much difference between these two things and "the pairs". Worth noting that our csv paired format doesn't preserve the time-series shape of the pairs, unlike the protobuf, but also that we don't message the pairs as protobufs yet, so we are stuck with csv pairs off-to-the-side. But the only difference between these plots and the pairs is creating a graphic from them and creating graphics is not core business. Hmmm.
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2020-12-08T11:50:10Z
As author of the ticket, I feel like I should reply, though James largely answered for me. :)
Yes, we need to have a means by which our users can see a forecast progression plot or scatter plot of the pairs. How that happens, I don't know, but the requirement is there. Thanks,
Hank
Original Redmine Comment Author Name: James (James) Original Date: 2020-12-08T12:05:41Z
Hank wrote:
As author of the ticket, I feel like I should reply, though James largely answered for me. :)
Yes, we need to have a means by which our users can see a forecast progression plot or scatter plot of the pairs. How that happens, I don't know, but the requirement is there. Thanks,
Hank
If we were to add this to the graphics app, then it would require us to turn on the messaging of pairs first (which would, in turn, require making the pairs writer like any other message consumer, rather than something off to the side, plus it would need the new charting methods in the graphics app that consume the pairs). Regardless, the work to message the pairs needs a ticket - will add that.
In summary, how it happens:
Original Redmine Comment Author Name: Chris (Chris) Original Date: 2020-12-08T14:03:11Z
Are we pretty much just talking about what the ISEDHydroGrapher spits out? If so... maybe we sit this one out. This sounds like feature bloat and it might be a better idea to have dedicated software for it rather than trying to bolt it on to an evaluation engine.
Would they be nice to have in the same place? No doubt; that adds a good bit of context.
Original Redmine Comment Author Name: James (James) Original Date: 2020-12-08T14:09:34Z
They're similar, I think, at least the progression plots (not the scatter plots). I guess the main difference is paired vs. unpaired and, more precisely, the same pairs that are used to compute statistics.
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2020-12-08T14:12:41Z
Just saw James's response, but posting anyway since I spent the time to write it. :)
Chris wrote:
Are we pretty much just talking about what the ISEDHydroGrapher spits out?
Similar. I will say that scatter plot has been mentioned, as well, and that is not covered by ISEDHydroGrapher.
If so... maybe we sit this one out. This sounds like feature bloat and it might be a better idea to have dedicated software for it rather than trying to bolt it on to an evaluation engine.
Would they be nice to have in the same place? No doubt; that adds a good bit of context.
I agree that this is not necessarily a high priority, because other tools are filling in the gaps. To be clear, however, this would tie the forecast progression plot to an evaluation. It would provide the time series for only pairs contributing to calculating metrics and after rescaling the data to the @desiredTimeScale@ (correct me if wrong). Presumably it would/could also be related to the pools of pairs, though that may lead to some funky plots. The ISEDHydroGrapher is more of a forecast analytics tool, and is not tied to an evaluation.
Thanks,
Hank
Original Redmine Comment Author Name: Chris (Chris) Original Date: 2020-12-08T15:04:39Z
I was just using the hydrographer as an example; I've seen three or four tools spitting out progression plots which had me wondering.
Regardless of what is planned for these sorts of plots, I agree that being able to pump out pair details as messages is a great start.
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-10-05T17:56:06Z
This is the development ticket for a forecast progression plot. It was discussed in the context of the challenges slide (interrogating raw data) from Seann presented during the 6th Users' Forum. James mentioned perhaps outputting it as HTML to allow for zooming, since it is likely to include a bunch of data spanning a long time period.
Hank
Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-10-05T20:13:51Z
James mentioned perhaps outputting it as HTML to allow for zooming, since it is likely to include a bunch of data spanning a long time period.
Chris saw this comment and pointed me to the ISEDHydroGrapher. It creates an HTML file that contains all of the data needed to visualize a progression plot using Plotly. I've attached an example (gzipped).
When I mentioned figuring out how to produce something like that in Java, he responded that it could be a Python microservice behind the COWRES. I'm picturing something similar to how the current events broker and graphics client works, except with the client being Python code that generates Plotly HTML that is portable and ready to be displayed. Not sure if it could use the same events broker, however: that is used for messaging statistics, not raw data, but the developers can figure that out.
Anyway, I think this proves the concept, and using a microservice architecture makes sense, its just a matter of identifying tooling.
Thanks,
Hank
P.S. ISEDHydroGrapher is in GitHub here:
https://github.com/NOAA-OWP/ISEDHydroGrapher
Here is some code that talks to the ISEDHydroGrapher running on ised-dev1 (domain omitted) and prepares the self contained HTML:
>>> import requests
>>> url = "http://nwcal-ised-dev1.[domain]:9773/nwm/hydrograph?lid=DRRC2&observations=on&thresholds=off&navigate=off&configuration=short_range&variable=streamflow&date_range=[PT10H,now]&colors=ed0909,0c5eed&format=text"
>>> response = requests.get(url)
>>> with open('./test_file.html', 'w') as test_file:
... test_file.write(response.text)
If you use a browser to visit the endpoint above, you won't just be able to download the HTML. Chris had to use Python to get a full self-container HTML.
Original Redmine Comment Author Name: James (James) Original Date: 2023-10-05T20:28:15Z
Makes sense to me. Yes, it could (should) using the same messaging api. There is a placeholder for messaging the pairs (a separate ticket) and we can message the raw data too, no problem. Good excuse for developing the first python client - after that, we'll probably want to move the netcdf writing to python.
Author Name: Hank (Hank) Original Redmine Issue: 45682, https://vlab.noaa.gov/redmine/issues/45682 Original Date: 2018-01-19
A forecast progression plot merely displays forecast time series by T0 (i.e., T0 in the legend) in a time series plot with observations displayed. Alex has some examples for the IOEP, which I'll attach later when we start to work on this.
Most forecast systems, including CHPS/FEWS, can generate forecast progression plots. Such plots, when also displaying observations, will likely show all of the observations and forecasts, not just those limited to pairs, and at their original time scale, though they may include features to adjust that scale. In other words, each time series will be loaded independently of the others and then displayed.
For the WRES forecast progression plot, based on an email exchange with James, I propose that we include only paired data in an evaluation version of the forecast progression plot. Thus, the plot would reflect the time series of pairs handed off to WRES for the pooled windows and in the desired time scale of the WRES output (i.e., after aggregating).
Obviously, the exact design for the plot is up for debate and I'm certainly open to other ideas.
Scheduled this for SR1 and prioritiy normal, reflecting that this may be slipped if we don't have time to do it. I've added Alex and James as watchers for now.
Hank
Redmine related issue(s): 118363