Issue with Fog Icon With Precipitation

[cloneofghosts]

I had posted this comment in the Weather Details thread but thought creating a new issue would be better. Are general API issues going in here or do you have a specific repo for it?

I just noticed that Fog seems to be overriding the rain icon as seen when you query here https://dev.pirateweather.net/forecast/[API]/44.6476,-63.5728?units=ca.

"time": 1648245600,
"icon": "fog",
"summary": "Fog",
"precipIntensity": 0.3549,

Shouldn't that be showing the rain/snow/sleet icon instead of fog? The hierarchy should be:

Rain/Snow/Sleet Fog Wind Clear/Partly Cloudy/Cloudy

Also the icon is still broken on the front-end viewer:

[alexander0042]

This is the best spot for issues, thanks for moving it over here!

Definitely need to figure out that fog icon now that the API is returning it from time to time, so I'll take a look at that shortly. I think the icon is just missing, but need to double check.

As for the order, that is how it's working now. I've added a documentation tag to remind me to document this, but the exact logic is:

Hourly:

1. If probability is greater than 40% and precipitation is greater than 0.25 mm, then precipitation.
2. If visibility is less than 1 km and it's during daylight hours, then fog
3. If winds are greater than 10 m/s, then wind (open to suggestions if anyone has a more specific number)
4. If cloud cover is greater than 80%, then cloudy
5. If cloud cover is greater than 50% and less than 80%, then partly cloudy
6. If cloud cover is less than 50%, then clear

Daily:

1. If max probability is greater than 40% in any hour and total precipitation is greater than 1 mm, then precipitation. Type is based on the most common (modal) precipitation type.
2. If average visibility is less than 1 km, then fog
3. If average wind speed is greater than 10 m/s, then windy
4. If average cloud cover is greater than 80%, then cloudy
5. If average cloud cover is greater than 50% and less than 80%, then partly cloudy
6. If average cloud cover is less than 50%, then clear

I remember writing these thinking, "wow there are a lot of assumptions that need to go into a weather API", so it's good to get around to documenting and getting feedback on these details. Initially, I didn't put much thought into the icon field, since I assumed the raw data would be more useful, but that hasn't been the case at all.

I'm thinking what happened here was the probability was less than 40%. Maybe I should use 30% as a threshold instead, or some sort of combination of probability and amount together (i.e. probability * amount > 1)?

[cloneofghosts]

Definitely need to figure out that fog icon now that the API is returning it from time to time, so I'll take a look at that shortly. I think the icon is just missing, but need to double check.

I know you updated it in the repo but maybe you forgot to update it wherever it's being hosted?

Hourly:

2. If visibility is less than 1 km and it's during daylight hours, then fog

Is there a particular reason as to why it only applies in the daytime and not all the time?

3. If winds are greater than 10 m/s, then wind (open to suggestions if anyone has a more specific number)

I think that sounds reasonable.

4. If cloud cover is greater than 80%, then cloudy
5. If cloud cover is greater than 50% and less than 80%, then partly cloudy
6. If cloud cover is less than 50%, then clear

From what I've seen with Dark Sky I think they show the partly cloudy icon between 30%-85% cloud cover but I think it can be adjusted. I know that NOAA has some definitions that might be useful here? I can share them if you would like to see?

I'm thinking what happened here was the probability was less than 40%. Maybe I should use 30% as a threshold instead, or some sort of combination of probability and amount together (i.e. probability * amount > 1)?

I didn't look at the precip probability to see what it was so it definitely is a possibility that it was lower then 40%. For pop I think most sites use 30% as a threshold to show a precip icon if there is any forecasted.

Sort of unrelated but I feel like there should be some sort of visibility cap so things are consistent everywhere. I've seen with HRRR that you can get 80+km visibilities and when you compare it to GEFS it seems a little weird. I know Dark Sky caps it out at 16 and I know other places use 24/16/10 as their cap. It might make sense to cap it at 24.14 so it's consistent with GEFS? I don't know what the max is for NBM but it might make sense to keep things consistent everywhere?

[alexander0042]

Yea- I definitely added the fog icon in, so for whatever reason the s3 bucket didn't update! I'll recompile it and try uploading again, which should resolve that!

Reason it's only during daylight hours is because the HRRR model returns 0 km for visibility at night, so I can't tell if it's foggy or not using that approach. There might be another variable in there somewhere, but since fog typically occurs during the day, it's not really an issue.

If you have seen NOAA definitions somewhere for cloud cover, I'd love to see them! I try to follow "standards" as much as I can here, so that's a great approach.

Also think 30% makes more sense, so I'm going to drop it to that. Good about about capping it as well, since it loses relevance beyond 24.14. I don't want people to see a higher number and think it means something different than what GEFS returns

[cloneofghosts]

Reason it's only during daylight hours is because the HRRR model returns 0 km for visibility at night, so I can't tell if it's foggy or not using that approach. There might be another variable in there somewhere, but since fog typically occurs during the day, it's not really an issue.

I did a quick query for my location and this is one of the entries in the hourly section and it seems to have a proper visibility variable so not sure where it's always returning 0. Maybe it was foggy that night when you checked?

"time": 1648537200,
"icon": "clear-night",
"summary": "Clear",
"precipIntensity": 0.002,
"precipProbability": 0,
"precipIntensityError": 0.003,
"precipAccumulation": 0.002,
"precipType": "snow",
"temperature": -8.71,
"apparentTemperature": -14.96,
"dewPoint": -16.21,
"humidity": 0.53,
"pressure": 1008.6,
"windSpeed": 14.19,
"windGust": 31.5,
"windBearing": 297.38,
"cloudCover": 0.5,
"uvIndex": 0,
"visibility": 35.5,
"ozone": 382.17

If you have seen NOAA definitions somewhere for cloud cover, I'd love to see them! I try to follow "standards" as much as I can here, so that's a great approach.

This is what I could find from NOAA:

Mostly Clear When the 1/8th to 2/8ths of the sky is covered by with opaque (not transparent) clouds.

Partly Cloudy Between 3/8 and 5/8 of the sky is covered by clouds.

Mostly Cloudy When the 6/8th to 7/8ths of the sky is covered by with opaque (not transparent) clouds.

Cloudy When 7/8ths or more of the sky is covered by clouds.

I couldn't find anything for Sunny/Clear but I would assume less then 1/8.

[alexander0042]

Yea, as I typed that out about visibility I was wondering if that made any sense. I honestly can't figure out why I was limiting fog to during the day, so I'm going to remove that condition! Night fog can happen (maybe it was when I was setting it up?), no point in limiting it. The change is live in the .dev back-end, and I'll push it to prod later this week.

And thanks for sharing those definitions- can you pass along the URL, since I'll add it to the docs. The plan is to use:

• < 3/8 (0 - 0.375): Clear
• 3/8 - 5/8 (0.376 - 0.625): Partly cloudy
• `> 5/8 (0.626 - 1): Cloudy

Also, makes a lot of sense to switch the precipitation threshold to 30%, so that's done and in .dev!

[cloneofghosts]

This is the article which has links to the NOAA glossary which is where I found the definitions https://weather.com/science/weather-explainers/news/common-weather-terms-used-incorrectly. It has a link to the NOAA glossary included but if you want direct links https://forecast.weather.gov/glossary.php?word=cloudy and https://forecast.weather.gov/glossary.php?word=clear

I think because Dark Sky is limited for icons it might make more sense to use 6/8 (75%) as the switchover to Cloudy instead of 5/8 but you can leave it as is if you prefer.

[alexander0042]

I know I've said this before, but this whole discussion exemplifies the value in having an open and documented weather API is important! What icon is returned matters, but is also something that isn't nearly as cut and dry as it seems. I think 75% makes sense as well for cloudy, so I'll set it to that!

[cloneofghosts]

I think after that change the icons make more sense as I always found it weird that clear went up to 50% cloud cover. I noticed that the visibility is capped at 16 instead of 24. Both are fine just curious if there was any particular reason for it.

EDIT: I see that the daily block was missed so that will need to be updated.

I was thinking in the future you could always add an optional parameter for an extended icon set for those that would like it when you implement the text summaries.

3. If winds are greater than 10 m/s, then wind (open to suggestions if anyone has a more specific number)

I think the wind icon doesn't work correctly as when I look in the hourly details I see this (cut down for spacing):

"time": 1648767600,
"icon": "clear-day",
"summary": "Clear",
"precipIntensity": 0.1971,
"precipProbability": 0.43,
"precipAccumulation": 0.0197,
"windSpeed": 39.39,
"windGust": 78.3,
"cloudCover": 0,
"visibility": 16.09,

Shouldn't that show as windy instead of clear here? 10 m/s is 36 km/h so it should be showing the wind icon and not clear and on a test last night it didn't show in current conditions either.

[alexander0042]

Two steps forward, one step back! Something is still going on with the wind icon, so looking into that now; however, the fog icon is working, so that's something:

Turns out I forgot to upload the new index.html file to the s3 bucket, so even though the new java script files were there, they weren't being used. Hopefully no one noticed, but since it wasn't foggy anywhere, I had to make the dev backend return fog for all locations for 5 minutes to test this

[cloneofghosts]

I think most people aren't using the dev backend so it likely didn't affect things too much. Did you see my note about the visibility not being capped in the daily blocks? It is everywhere else so it just got missed when you were setting that up.

If you need a city to test with you can use Brocket, AB, Canada as it's showing a wind speed of 44 km/h currently.

[alexander0042]

Fixed and fixed! Wind was another (and hopefully the last) unit issue. About 20% of the way through designing this, I decided to use SI units until the very last step, when I adjust for whatever the requested units are. However, I used km/h for wind, which messed up the icon threshold. Corrected now in the dev and webpage, which relies on the .dev back-end. Also just missed the visibility cap, so that's corrected as well!

[cloneofghosts]

Awesome and I see that the Clear/Partly Cloudy/Cloudy changes have been implemented as well. I feel like it should say Windy instead of Wind for the wind icon since it seems weird that it just says Wind.

Otherwise everything here is done. Do you need to keep this issue open to remind you to do the Documentation or do you have it noted somewhere else?

[alexander0042]

I agree that windy would be more consistent, but the Dark Sky docs say it's wind, so that's what applications expect. I did change the summary text to 'Windy', which is a little clearer.

Docs have been updated https://docs.pirateweather.net/en/latest/API/#icon! Still a lot of work to do on these, but this is a start. That means that we can go ahead and close this issue and keep moving forward, on parameter at a time!

[cloneofghosts]

Yeah, I meant the summary text and not the icon. Since the documentation is updated I'll close this issue.

[cloneofghosts]

Did you forget to launch 1.1.6? The public API is still on 1.1.5 and I wonder if you've forgotten to make it live.

[alexander0042]

Oops! You're 100% correct- I forgot to hit the "Deploy API" button! It's live now, and thanks for noticing it 💨

[TheEPIX]

Great work!! Just one idea to provide a smooth change from darksky: could you please support the icons on your URL? (http://pirateweather.net/images/weather-icons/...)

[cloneofghosts]

@TheEPIX I'm not entirely sure what you are looking for here. Are you asking if PW can host their own icon set for you to use or do you want PW to host the old DarkSky icons for you to use?

If it's the latter then I don't think it would happen because only their SkyIcons were made publicly available and I doubt that @alexander0042 would want to risk hosting the newer set of icons. There are plenty of icon sets out there that are publicly available that you can use instead.

[TheEPIX]

You are right - i was asking for hosting a suitable PW owned iconset according to the icon-names used in the code...

[cloneofghosts]

@TheEPIX As far as I'm aware there is no PW owned iconset but you can always create your own issue to see if the dev can get one created or if there's one that he would recommend for you.