Displaying model error

[daviswx]

Model error metrics are now being generated, and model error will probably need to be displayed in the DSS in some form. Doing this in an intuitive and not overly complicated way is our main challenge.

Since the model error will have three different maps -- a mean and ±2 std. devs -- we could show these on the three-map layout and have a separate map display option in the menu just for model error. However, @hadinon and I both feel like that won't be a useful way to view the error data or make assessments using it.

Instead, we discussed having the error data overlaid on the maps and time series to better show which locations have values within the spread of model error, which would render them less meaningful.

My idea is to have a single "Show Error" button/option on the page. When clicked, it would overlay a semi-transparent (the opacity could be changed with a slider bar) layer on each map, and locations with values within the model error would be more opaque (possibly in gray?).

The time series would have either three horizontal lines (one solid, two dashed) corresponding to the model error, or a shaded underlay (if it's possible in Highcharts) from 0 (on the Change display) or the historical average to the extent of the model error.

This approach would require additional processing time to create grids with binary values for each point -- whether it's inside or outside of the range of model error -- for every RCP and time period. However, IMO it would be a much more meaningful way to visualize the error.

In issue #88, we discussed using a similar approach of changing the shading/opacity and effectively hiding points within the model error for the seedling deployment tools.

[hadinon]

The following error metrics have been generated for all thresholds (32F, 28F, 25F, 20F, and 15F) within the Extreme Min Temp tool:

• multi-model mean bias (center map) [[METDATA observed - model baseline]]
• multi-model mean bias +/- 2 stddev of bias (left & right maps)

Reference my email for a link to the final files that are in THREDDS. I'm working on the rest of the tools now and will keep you posted.

In the meantime, Ryan mentioned wanting to see the 3-map layout of error so maybe you could mock that up first with a separate map display option in the menu just for model error (as mentioned above)?

[hadinon]

Those same error metrics have also been generated for summer precip, summer temp, and summer dryness index. I'll email you the links.

For the seedling deployment tool grids (aka. coldest day of the year), I have to regenerate 1,120 of the model baseline files before proceeding with the error calculations. These were deleted back in July 2015 when convection was running out of space. Anyways, that script is running and is estimated to take two hours. I'm hoping to produce the error metrics for this tool today as well but it may be Monday morning depending on how bad convection is getting beat up right now. :wink:

[daviswx]

Here's what the tool might look like with the model error displayed. Let me know if any of the display names or map titles should be changed before we show this to Ryan.

[daviswx]

Also, note that the color bars in the legend should be reversed. Red = higher positive values in this case.

[hadinon]

Noting this for completeness: the seedling deployment (aka coldest day of the year) error maps have been generated and I emailed the links.

As far as the map display options, I'm not sure if we'd have 3 different options if we decide to go this route above (3 map layout of the raw error metrics). We would probably just have one radio button that said "Model Error". If we decide to display error using the "if change < error" method, I'm not sure if we would need the 3 options either. We would probably do this second method as an overlay onto the existing 3 map layout. That said, I'm still trying to wrap my head around this displaying error concept.

As far as the map titles, I would change them to "Minimum Model Error", "Mean Model Error", and "Maximum Model Error" for the above display.

[daviswx]

Here's an updated version with those changes:

If we go with the other method (which I prefer), my idea was to have a button or switch somewhere on the page to show the model error. When it's turned on, it would show the model error overlays on the maps and time series, and a slider bar on the page would control the transparency/opacity of the map overlays.

[hadinon]

Quick note/reminder about this: once changes are implemented, remove this bullet on the DSS Introduction --> Time Series page: "Note: Model error is not currently shown in the DSS" and add another bullet indicating that model error is shown.

[hadinon]

I just talked to Aaron about this (not intentionally, it just came up when we were talking about something else entirely) and wanted to document it here before I forgot! What if we set (or let users select) some sort of threshold of tolerance with regards to the error. For example, they might say, "If the projected change is 2 degrees higher than the error, that will impact my operations so I will use this to make a decision (e.g. alter management practices) so please show the values. Otherwise, don't display the grid cells because the guidance is not useful for my altering decision making process". I would envision a few different thresholds for users to choose from, e.g. 0.5 degrees, 1.0 degrees, and maybe 2.0 degrees for temperature-based tools. So, this would mean generating many more error grids (so maybe PINEMAPv2 worthy!) but is a neat idea. Overall, it would be a way to incorporate risk but would add yet another level of complexity for users to interpret.

[daviswx]

Preface: I know we talked about this today, but I'm adding it here so I don't forget over the weekend. :wink:

A model error display is now available for the Minimum Temperature Thresholds tool. For now, only data for RCP 4.5, the 2060-2079 time period, and the 32°F threshold is shown; those are hard-coded in since they're the only binary error files available at the moment.

Based on this display and what we've talked about in this thread and elsewhere, here are some questions or topics for discussion:

• [x] Let's check to make sure we want to be using the '1' values as a mask and making the '0' values transparent. I think this is correct, but it's worth double-checking.
• [x] We should make sure that the correct error layer is showing on each side map. For this tool, I think the error layers should be flipped for the Projected Change display -- i.e., the layer with more area masked out should be on the right (?) side map, while it's on the left (?) for the Projected Average display -- because of the negative change deal that we've discussed ad nauseum.
• [x] I'm currently using white for the mask color, but this can be problematic for maps where one of the background colors is also white. The Projected Change map is a good example of this. I know we discussed using light gray for the mask color, so I will play around with the color ramp to see how this looks.
• [x] When the model error is turned on, I have one line in the legend noting this, but otherwise nothing else changes to tell the user that model error is being displayed. A more drastic change -- and one that I can see Ryan suggesting -- might be to change the map title bar colors and add a note in there as well. I'm of the opinion that reducing the amount of visual goodies (especially text) thrown at the user is better, and if a user checks the "Show Model Error" box, they're probably expecting to see exactly that. But I will admit that the current one-line explanation might be a bit minimalistic.
• [x] We need to figure out what to do when the selected location is within one of these masked (error > projection) regions. We discussed hiding it on that map, although that might make people think that the map is broken. I also mentioned possibly adding some text beneath the marker indicating that it's within the bounds of the model error, although concisely stating this in a few words might be tricky. It's worth noting that all 3 values for a location will still be visible on the time series, so if our goal here is to hide the data because it's not meaningful, then we'd need to do the same thing on the time series, and I recall Ryan saying that he wanted all values visible there.
• [x] Will there be any model error metrics for the historical map? If not, we might want to disable the "Show Model Error" box when the Historical Average display is shown.
• [x] The model error still needs to be added to the time series. To do this, I will have to query the actual error files (not the binary/mask versions) via an OPeNDAP call for the selected location and pull out the mean and ± 2 standard deviation values. I can then add horizontal lines to the time series plot using the Highcharts plotLines feature, as this example shows.
• [x] At some point, we need to draft model error explanations for the tooltips and the FAQ page.

[daviswx]

Update to one previous point: I have changed the mask color to light gray so it's distinguishable from the background map and any white colors in the data.

[hadinon]

Here are my responses (some of which we've discussed in person but adding here for completeness):

• Yes, it seems like the data are being displayed properly! The flag will have a value of 1.0 where the error exceeds the change and a value of 0.0 elsewhere. So, we want the original grid to show for the 0.0 grid cells and the masked layer to show for the 1.0 grid cells.
• Agreed, let's double check this. Can you remind me which netCDF is being displayed on the left and right maps as far as the original file, e.g. 2stddev_plus_mean_rcp45_2060_2079.nc or 2stddev_minus_mean_rcp45_2060_2079.nc? Similarly, which netCDF is being displayed on the left and right maps as far as the masked grids?
• The light gray color looks good.
• Your "one-liner" about the model error jumps out which IMO is good. However, I might suggest a slight wording tweak to: "For areas masked in light gray, future changes are smaller than the climate model error." -- does that sound OK?
• I am in favor of hiding the values on the time series, but do recall Ryan saying he wanted all values visible there so perhaps a discussion item for our weekly PINEMAP meeting. As far as how to handle what happens when a user clicks within a "masked" area, could we say "N/A (error exceeds projection)" or would that be too much text?
• Correct, there are no model error metrics for the historical map. So, we will want to disable the "Show Model Error" box if the historical average is being displayed.
• Sounds/looks good to me.
• I'll go ahead and add an FAQ for the model error to our working doc (which I still need to revisit BTW -- apologies). Potential wording for a tooltip: "Model error is a metric to determine how well the downscaled global climate models (GCMs) perform. If the error exceeds the future projected changes, the data are not as useful for decision-making. Thus, in those cases, DSS values are masked out with a light gray color."

[daviswx]

Here are the files being used on the Minimum Temperature Thresholds maps:

Projected Change Left map: 2stddev_plus_mean Middle map: mean Right map: 2stddev_minus_mean

Projected Average Left map: 2stddev_minus_mean Middle map: mean Right map: 2stddev_plus_mean

Model Error (currently) Right map: 2stddev_minus_mean Middle map: mean Right map: 2stddev_plus_mean

Because of this, I think I need to flip the error overlays when they're displayed on the Projected Change maps.

[daviswx]

Also, I have converted my previous post into a task list so it's easier to track which questions or issues we've resolved.

[hadinon]

OK, so it sounds like the Projected Change error maps need to be flipped but they are OK for the Projected Average error maps. Is that possible to easily do in your code? For the Projected Average, I think we will want to use the same error maps as Projected Change but let's double check with Ryan today.

Wow, that's awesome about the task list -- great idea!!

[hadinon]

One liner: "For areas masked in light gray, historical model error exceeds likely future changes and may not be meaningful for some decisions."

[hadinon]

• Add meaning of color for error to the legend on map and time series.
• Put valid on map that says: "Error > Projection"

[daviswx]

I have fixed the error layers so they show on the correct side maps in the Min Temp Thresholds tool, updated the wording in the legend, and disabled the Model Error option on the Historical map. I also checked those items off the task list above.

I am currently working on getting the model error data from OPeNDAP calls and adding the lines to the time series. I do want to clear up something about this, though, and make sure I'm using the correct files.

For the raw model error values (not the binary flags), I am using these three files on convection:

Lowest Likely Change/-2 standard deviations:

pinemap/maca/extreme_mintemp/annual[THRESHOLD]error/error_final_bias_1986_2005/Avg_2stddev_minus_mean_bias_METDATA_obs_minus_MACA_baseline.nc

Multi-Model Mean:

pinemap/maca/extreme_mintemp/annual[THRESHOLD]error/error_final_bias_1986_2005/Avg_mean_bias_METDATA_obs_minus_MACA_baseline.nc

Highest Likely Change/+2 standard deviations:

pinemap/maca/extreme_mintemp/annual[THRESHOLD]error/error_final_bias_1986_2005/Avg_2stddev_plus_mean_bias_METDATA_obs_minus_MACA_baseline.nc

These files give values generally centered around zero, which is exactly what we need for the Projected Change time series. For the Projected Average time series, should I just add on the historical observed value to each?

For example, in the point I'm looking at in northern Georgia (34.1041°N, 84.1941°W), I get these values: Historical Observed: 61.5 days Model Error (-2 std. dev.): -8.9 days Model Error (mean): -2.4 days Model Error (+2 std. dev.): +4.0 days

So for the Projected Average time series, should the horizontal lines appear at 52.6 days (61.5 - 8.9), 59.1 days (61.5 - 2.4), and 65.5 days (61.5 + 4.0)? Or is it more complicated than this?

Also, should we show all three model error lines on the time series, or just the highest and lowest? I can't remember whether we've talked about this previously.

[daviswx]

Also, one more thought about the model error display on the time series. In Highcharts, along with doing lines across the plot, you can do full bands across the plot, like in this example (our's would obviously be horizontal, not vertical).

Here's a basic example of what it might look like in our time series:

The background color would probably need to be changed so it doesn't interfere with the bar color as much, but that gives a rough idea of a possible way to display model error. Personally, I like that better than the lines since people may not "read between the lines" and know what the space between them means.

[hadinon]

Yes, that is the correct approach for the Projected Average time series.

Showing the lowest and highest model error lines as full bands would be good.

Let me know if I missed any of your questions!

[daviswx]

Just to recap the latest developments, the error is being shown as a colored bar across the time series, so that's checked off of the to-do list above. However, there is a discrepancy between these values and the regions masked on the map since one uses the raw error data and the other uses the magnitude of the error.

Let's talk to Adrienne soon to see which method is best, and we can adjust the time series and/or map layer calculations accordingly.

Also, I still need to display the new marker text when the selected location is within the model error.

[hadinon]

OK, I had a lengthy conversation with Adrienne about this. In summary, I plan to recalculate the error as mean absolute error instead of mean bias. When generating the error mask / binary file, I will compare the mean absolute error with the mean absolute future difference/change.

• Error on the Maps: Overall, the logic for displaying the maps will not change.
• Error in the Time Series: We think it may be best to also do a "mask" concept on the time series data. For a particular 20-year period and RCP at a selected location, if any one of the three future change values is gray / masked out, we should not show that "bar" on the time series. Instead, add a note similar to what we add to the map: "Error > Projection".

I'll work on generating the new grids tomorrow. Let me know if you have any questions or would like to discuss more details in person.

[daviswx]

It's been a while since I had a statistics refresher, so off-hand, I'm not really clear about the difference between mean absolute error and mean bias. But I trust that you and Adrienne know what you're talking about!

I do have a bit of a concern about masking data in the time series, though. In my opinion, it's actually useful in that context to see exactly how the projections compare to the error.

For example, if the model error for number of days below freezing is from 50 to 60 degrees, it means one thing if the projected values run from 44 to 51 degrees -- barely within the model error -- and something completely different if the projected values run from 52 to 59 degrees -- fully within the error. It sounds like you'd want to treat these both the same way, and just remove the bars and stamp the error message in its place.

Also, having the error band across the entire time series can essentially function as a mask (with changeable transparency to boot) without having to hide entire bars altogether. One thing I might try to change is moving the error band in front of the bars, so when it's fully opaque, it does indeed hide the bars behind it.

I'm willing to listen if there is a compelling reason for removing the bars entirely if even part of them is within the model error bounds, but at this point, I think that's actually removing important information.

[hadinon]

I want to try to document all these error conversations in a summarized way so here are some notes:

• generate mean absolute error/bias only and use it for all 3 map outputs since the variability of the error does not equate to variability of projected changes
• for generating the binary flag files (used for showing the masked region on the maps), continue to compare mean absolute error/bias with absolute value of mean/mean+2stddev/mean-2stddev projected change (instead of generating mean absolute projected change [as well as mean+/-2stddev absolute projected change] under the assumption that all models have the same sign of projected change and thus, these two metrics are equal; however, this might get tricky with precipitation)
• for projected change time series, show +/- mean absolute error as full bands centered on 0
• for projected average time series, show +/- mean absolute error as full bands centered on the historical average value

Other notes:

• why mean absolute error instead of RMSE? RMSE would be very complex to explain to a professional forester audience
• see limitation above about using absolute value of mean/mean+2stddev/mean-2stddev projected change instead of mean absolute projected change [as well as mean+/-2stddev absolute projected change] for generating the binary flag files (used for showing the masked region on the maps)

Action item:

• double check to make sure that the masked map regions match up with the bars intersecting with the full bands of error shown on the time series (concern: time series is not showing absolute value of projected change but is showing +/- mean absolute error so might be OK but I need to run through an example to fully wrap my head around this :wink: )

Please reply with anything that I may have missed!

[daviswx]

This summarizes my understanding of our discussion as well.

Re: your concern in the final part, I think this will be fine on the Minimum Temperature Thresholds tool, but I'm not quite sure about the Summer Precipitation tool, which can have Projected Change values on either side of 0. Luckily, we don't have to worry about that quite yet.

On the DSS, I will remove the ±2 std. dev. error files and instead bound the time series error band with the mean value on either side of 0. I may also add dashed lines at the top and bottom like on our sketches.

[hadinon]

Posting an update here for good measure: the binary flag file has been regenerated using mean absolute error and comparing that with the absolute value of the mean projected change values for RCP4.5 2060-2079 on the Minimum Temperature Threshold 32F. This binary file will now be used for the error mask on the center map of the DSS. Binary flags have also been generated for the side maps by comparing the mean absolute error with the absolute value of the mean+/-2stddev project changes.

I'm working on generating these same type of binary files for other RCP and time slices for 32F threshold.

Also, for the time series, the raw mean absolute error file has been generated (same name as previous mean bias error file).

[daviswx]

I have checked off the final two to-do boxes above.

When the model error is enabled and a clicked location has a value less than the magnitude of the model error, a message displays (in red) above the map marker stating "Error Exceeds Projection".

Also, I added the preliminary Q&A about model error to the FAQ box, although we still need to write a part about the error band on the time series.

Once the binary files for the other RCPs, time periods, and thresholds are ready, I can comment out the hard-coded RCP 4.5 and 2060-2069 period so the page will dynamically update the error overlays.

[daviswx]

Based on feedback and discussions at the annual meeting, here are some suggested action items related to the model error:

• Leslie suggested showing the model error by default and removing the ability to turn it off or even turn down the transparency. After talking about it with Ryan, it seems like a reasonable idea, but we want to think about it some more and potentially run it by the rest of Aim 6.
• In our terminology throughout the DSS, refer to the model error as "model limitations" since "error" may imply an error on the page or loading the data.
• Don't use red coloring for the model error/limitations and descriptions, like in the legend. Again, to some users, red may imply an actual error on the page.
• The disclaimer in the legend could be changed to say something like "Caution: For areas masked in gray, values are not meaningful given model limitations".
• When a clicked point is within the model limitations, change the label to say something like "N/A" or "Data Not Applicable". Don't show this in red text like we are now.
• In the time series, use two bars for the historical time period: the observed value (in blue) like we have now, and the spread of the error in a different color. The horizontal lines and band can be changed from light red to this new color. Evan suggested that white may work and wouldn't distort the background color values like the red does.

Also, I found a couple small bugs with the current implementation of model error:

• If a location was selected and the model error overlays are then turned on, the location box flashed green several times. Note that this may be only for Macs since I can't replicate it in Chrome and I only remember seeing it on Ryan's laptop and the iMac during demonstrations.
• If a location is selected with values within the model error (and their map marker labels change to "Error Exceeds Projection"), then the model error is turned off, those labels don't change back to the actual values.

Both of these bugs may not be problems if we decide to show the model error at all times, but I'd like to fix them before we move to that step.

[hadinon]

I was able to replicate the bug related to the location box flashing in green by using Firefox but not in Chrome.

[daviswx]

The multiple flashes bug with the location box in Firefox seems to be fixed. My call to a function that updates the location was embedded in a loop, so it was getting triggered 3 or 4 times instead of just once.

After moving this out of the loop, it also fixed the second bug mentioned above with the marker labels not updating when the model error is turned off.

[daviswx]

I have made some progress on the action items listed above:

• The binary masks now display by default on the Projected Change and Projected Average maps on tools for which they are available. The selection box at the top right corner of the page with the checkbox and transparency slider is now gone, so the main cue that model error is being displayed is the note in the legend above the maps.
• In that legend item, model error is now referred to as "model limitations". Also, the text disclaimer was changed to the suggested wording.
• If a clicked point is within the model limitations, the text above the map marker says "Within model limitations". I decided to go with this instead of "N/A" or "Not Applicable" since I thought those might be confusing or not easily understood.
• An additional bar has been added to the time series showing the range of model limitations. Also, the background band that stretches across the entire time series has been changed to a very light gray color. Note that this bar only displays on the Projected Change and Projected Average displays; I figured since we aren't showing model error on the Historical Observed map, for consistency's sake, we might not want to show it on the time series below that map.
• The red coloring in the legend item and time series bars/lines has been changed to orange, per Ryan's suggestion.

We still have a few action items related to model error/limitations:

• [ ] Write tooltips -- one for the map legend, one for the time series legend -- and FAQ box item(s).
• [ ] Generate the binary overlay and raw error files for all other thresholds and tools (I can't remember off-hand which of these are done yet)

And maybe some other things that I'm forgetting?

[daviswx]

The collective SCO group has suggested changing the map marker wording again since "Within model limitations" may be conceived as a good thing. Off-hand, I don't have any different ideas. Ryan suggested "Exceeds model capabilities", but I also wonder if users might interpret that as a good thing since it includes the word "exceeds".

[hadinon]

Ryan suggests: "Exceeds model capability" and "Outside margin of error". Feedback from the PINEMAP meeting suggested avoiding the use of "error" so maybe the latter is out. I just noticed your comment about "exceeds" possibly being interpreted as a good thing. Perhaps "Outside model capability"? Or, "low confidence"?

[hadinon]

Also, my only suggestion about the most recent changes is that the orange "Model limitations" text on the time series is sometimes hard to read depending on the background colors of the graph. Perhaps try white text or add an outline around the orange text like this?

[hadinon]

I asked Kieran if she had any ideas. Here's what she came up with (using a rule of 5 words or less):

• "Exceeds model capabilities"
• "Beyond model capabilities"
• "Not within model capabilities"

Or, since people will be basing decisions off of these models, you could call these areas a region of:

• "Model unreliability"
• "Model uncertainty"
• "Model inconclusiveness"

Also, "Unable to predict with current model" is very straightforward, but too long :(

[daviswx]

I had the same thought about the "Model limitations" label being tough to read. However, Highcharts styling doesn't support a text border like you suggested, and using a drop shadow isn't much easier to read and also apparently isn't supported well in IE.

I did just darken up that text color so hopefully it will stand out a bit more:

I don't necessarily have any ideas about the marker wording, but in my opinion, any of the ones that include terms like "exceeds", "beyond", and even "capabilities" are likely to be misinterpreted because those are all positive terms.

[hadinon]

Thanks for the update about the model limitations label.

At our meeting today, did we get consensus on which terminology to use for the marker? We've already asked Aim 6 for guidance. Does anyone else come to mind that might be able to provide assistance with this? Maybe Tim?

[daviswx]

Ryan suggested saying something like "Outside margin of error". Although that still has the potential problem of using the word "error", I think it's one of the better ideas we've come up with so far.

The Aim6 folks, particularly Gary, Eric, and Leslie, were fairly helpful with crafting this sort of language at the annual meeting, so maybe they would be good ones to consult with. I'd suggest that we might want to come up with a few additional possible wordings to run by them, and also allow them to suggest their own.

[daviswx]

When the model error is displayed in the time series, note in the tooltips when mousing over the bars whether values are within the range of model error. For consistency's sake, use the same wording as whatever we decide for the map marker labels ("outside margin of error"?).

[daviswx]

Here are the wording options we'd like to run by Aim6:

Outside margin of error Within model limitations Exceeds model capabilities Data not applicable Model not applicable Error exceeds projection Limited model skill Other: ___

[hadinon]

A Google Form (to be sent to Aim 6 soon) has been created with those wording options and another question about the respondent's role on the project, e.g., co-PI, staff, student. I thought it might be neat to stratify responses by role.

[daviswx]

For now, I have changed these labels to read "Error exceeds projection". Once we get the feedback from Aim 6 et al., this can be updated to reflect the consensus suggestion.

[daviswx]

Just to summarize the feedback we received, the PINEMAP respondents were split in their preferences, but three answers received the most support: "Error exceeds projection", "Limited model skill", and the volunteered response of "Low/limited model confidence". Direct feedback from Aim 6 suggested that they also preferred "Data not applicable".

After further discussions with Aim 6, we decided that each of these had drawbacks, so we instead went with a slightly longer option, but one that more closely reflected what we already had in the legend: "This location's value is not meaningful".

The new wording has been implemented on the PINEMAP DSS and Climate Voyager development sites and will be pushed to the public-facing sites in the next updates.

I have also been drafting some ideas for the model limitations tooltip. It's tough because there is limited room to work with (I'm trying to limit it to 3 or 4 lines) an a lot to explain! Of course, we do have the FAQ to link to at the bottom of the tooltip. Here are a few possible options I have come up with:

• In areas where the model baseline data does not closely match historical observations, we lack confidence in the future projections and thus mask those areas on the maps below.
  • This one may introduce an unfamiliar term -- model baseline -- and it doesn't explicitly state the criteria for masking an area (error > projection), although I don't know whether we need to say that in this tooltip.
• By comparing model performance to historical observations over a time period in the past, we can see how well the models performed at each location. In areas with poor historical performance, we lack confidence in the future projections.
  • This one avoids unfamiliar terminology but still doesn't state the criteria for masking areas on the maps.
• To see how well the climate models capture the known climate of an area, we can compare the model baseline data over a past time period against historical observations. If this difference exceeds the projected future changes, we lack confidence in the projections and thus mask those areas on the maps below.
  • This one again introduces the term "baseline", and it's a bit too long now at 6 full lines.

Let me know if you have any other ideas for how to word this.

[hadinon]

The second one seems like a good balance. I've added one additional word in there! Thoughts?

"By comparing model performance to historical observations over a time period in the recent past, we can see how well the models performed at each location. In areas with poor historical performance, we lack confidence in the future projections."

[daviswx]

I made one more small wording change before adding this to the DSS and Climate Voyager:

"By comparing historical observations with model performance over a time period in the recent past, we can see how well the models captured those past conditions each location. In areas with poor historical performance, we lack confidence in the future projections."

That has been implemented so I will close this issue for now. We may need to re-open it once other error grids are ready.

[hadinon]

I'm not going to re-open because this is minor -- should it be "...those past conditions at each location"?

[hadinon]

I was looking through the above comments and there are a few outstanding items:

a) We discussed adding a tooltip about model limitations for the time series legend. I'm thinking the same text as the map tooltip could be used. Thoughts?

b) From above: "When the model error is displayed in the time series, note in the tooltips when mousing over the bars whether values are within the range of model error. For consistency's sake, use the same wording as whatever we decide for the map marker labels."

c) Generate the binary overlay for all other tools.

I'm still working on c).

[daviswx]

I have added that missing word in the map legend tooltip and added a model limitations tooltip in the time series legend. The text isn't exactly the same since I wanted the time series one to specifically mention the meaning of the range and values that fall within it.

I can definitely add a note in the time series tooltips to indicate when values are within the range of model limitations. Do you have an idea for what this should look like? Here's a mockup that uses the same wording as in the map legend:

[hadinon]

Sounds good. I like the text that you used for the time series legend tooltip and the mockup looks good, too.

[daviswx]

The new time series tooltip text has been implemented on both the DSS and Climate Voyager development sites.