(UX research) Weather forecaster forecast validation

[jowand]

As a Weather Forecaster I want to be able to review my forecasts against actuals So That I can review the accuracy of my predictions and improve my forecasting

Additional Context

• we need to store forecasts in the context of the decision. How do forecasters want that context represented (e.g. representative weather pattern - choose our of X number of standard patterns, issue date, select features from a menu)
• forecasters are currently using a spreadsheet to track their forecasts, so entering their forecasts and then going back at a later date to enter what actually happened often gets put aside due to lack of time
• how do forecasters want to be able to review their work?
• what do they consider an 'accurate' forecast?

Acceptance Criteria

• [x] can describe how forecasters assess accuracy
• [x] identify weather features that forecasters want to mark down or note when storing their forecasts so that we can capture the decision making context
• [x] responses from 4/6 contacts

[PrakritiM]

Questionnaire:

https://app.zenhub.com/files/235861506/70356100-35e3-46ab-8e7c-f26dd86111e2/download

[PrakritiM]

User 1 response:

How are you currently storing/ maintaining/ tracking forecasts that you have made in the past? (i.e. in what format) I do not store ‘indice’ forecasts Video briefings are over-written each day I store weather warnings as PDFs on a government server I store spot forecasts as PDFs on a government server Regional forecasts are archived on the ‘fire weather page’

How often do you refer to/review these historical forecasts? For what purposes? Daily, for personal verification.

How many days in the past do you refer to analyse weather trends and patterns? Mostly past 48 hours, sometimes 3-10 days, very rarely 10+ days

How many days in the past do you refer to your historical forecasts for self-assessment? Sorry, I thought that’s what the last question asked?

What data do you use to validate your forecasts? Mostly weather station obs or field obs from fireline staff. A weather forecast is more than just numbers though, so satellite, radar & lightning data can all be used to verify a forecast.

How do you use this data to validate your current forecasts? …See how close the forecast was to what happened?

What is a ‘good’ forecast? It depends on the context. Generally temp within 2 degrees; RH within 3 percentage points; wind speed within 25% of value; wind direction within 45°. Lightning, cloud cover, and stability are harder to pin values to.

What is a ‘bad’ forecast? It depends. This is a judgement call. Sometimes a relatively small feature like a thunderstorm can lead to ‘bad’ local forecast values, but if there was only one thunderstorm in a 200km x 200km box, the forecast might not be that ‘bad’. The context is also very important – sometimes 5 percentage points on RH is a significant departure; sometimes not.

When you are forecasting weather values for the next few days, what other information would you want to record at the time to capture your decision-making context (weather pattern, key model issue date/time)? How do you want to be able to do this (e.g. drop-down menus, standard weather pattern selection)? Feel free to sketch this out ideal representation of your decision- making process. Interesting question… We are already pinched for time with the basic forecasting duties. I don’t see myself being very excited about also capturing the environment within which I made the decisions on the fly.

What do you consider an 'accurate' forecast for each of the following values (e.g. is there a range or standard deviation of the observed value is an 'accurate' forecast? Or is an accurate forecast more about identifying the trend in the weather pattern?) It really depends on the context. In some weather patterns +/- 10 degrees and +/- 20% is fine. For ‘peak’ fire season when certain other factors have been met, I would be happy with a forecast:

1. Temperature within 2 degrees
2. Humidity within 3 percentage points
3. Wind Direction within 45 degrees
4. Wind speed within 25% of value
5. Gust within 25% of value
6. Precipitation this is not a point value, but needs to include spatial coverage & character.
7. Lightning similar to precipitation, pretty tough to pin down.
8. Stability ditto.

[PrakritiM]

User 2 response https://app.zenhub.com/files/235861506/4261f396-f7d7-471b-9655-f9c0766ff543/download

User 3 response https://app.zenhub.com/files/235861506/16abca43-d9d6-45db-9ab4-7330fa545117/download

User 4 response https://app.zenhub.com/files/235861506/76f538d7-b44b-4da8-be93-14ff1370f0f1/download