sunshine and wind_gust_direction are NULL

[tawissus]

I wanted to work with sunshine data today. Unfortunately, sun and the wind direction are always NULL (eg in Berlin). Is it a bug or is it not available in some locations?

eg Berlin lat 52.54 / lon 13.54

[jdemaeyer]

Hi @tawissus,

while sunshine duration is indeed not measured at too many stations (and for your location in particular, the issue arises because Berlin-Tempelhof recently stopped recording it), it should still be available through Bright Sky. I.e. yes this is a bug, thanks for pointing this out! :)

I'm currently on vacation and won't have time to fix this for the next two weeks. As a work-around, you can supply dwd_station_id=00403 instead of lat and lon to the /weather endpoint. This will get you records from the Berlin-Dahlem station, which measures sunshine. (Here is the list of stations which record sunshine, you can find the DWD station ID in the first column and the date of the latest record in the third column.)

As a note to myself I'm sketching two approaches here:

1. Check how many stations actually have null sunshine values during nighttime and non-null values during daytime, maybe we can remove sunshine from the ignored missing fields setting?
2. Add a must_include parameter to /weather which allows (partially) overriding that setting

[tawissus]

Thank you very much. Have a nice vacation

[jdemaeyer]

Thanks! :)

1. Check how many stations actually have null sunshine values during nighttime and non-null values during daytime, maybe we can remove sunshine from the ignored missing fields setting?

postgres=# select * from (
postgres=#   select
postgres=#     source_id,
postgres=#     count(sunshine) as available,
postgres=#     count(*) - count(sunshine) as missing
postgres=#   from weather
postgres=#   where timestamp between '2022-11-04' and '2022-11-04 23:55'
postgres=#   group by source_id
postgres=# ) _
postgres=# where
postgres=#   available > 0 and
postgres=#   missing > 0
postgres=# ;

 source_id | available | missing
-----------+-----------+---------
(0 rows)

Looks like we can just go ahead and just drop sunshine from IGNORED_MISSING_FIELDS, particularly now that we have #129.

[jdemaeyer]

24c403c

[jdemaeyer]

@tawissus I've adjusted our mechanism which fills up incomplete weather records (like the ones from Berlin-Marzahn, which is the nearest station for your coordinates), and all regular lat/lon-based requests should now include values for sunshine again.

Missing values for wind_gust_direction stem from a different bug which I've added a ticket for here: #134

I'm closing this, but feel free to re-open if you think the issue with sunshine still exists :)

[jdemaeyer]

This change seems to have had a tangible impact on performance. I'll keep it in for now but am re-opening the issue to remind myself to look deeper into it once I'm back from vacation.

[tawissus]

Hey, thanks for your effort! I now have the sunshine in the JSON. But now please enjoy the vacation!

[jdemaeyer]

Thanks for the confirmation, I'm glad the fix worked :)

The slightly heightened average response time seems to have been a transient effect, so I'm closing this issue: