search

jdemaeyer / brightsky

JSON API for DWD's open weather data.
https://brightsky.dev/
MIT License
287 stars 18 forks source link

Worker process randomly dies during MOSMIX parsing #130

Closed smilingchemist closed 1 year ago

smilingchemist commented 2 years ago

Hi, thanks for providing this great API!

I have been trying around for some days now but I think there is an issue. During the past 48 hours (approx.) the field "last_record" shows "2022-08-08T16:00:00+00:00" in the forecast sources. Is there maybe a problem with updating the forecasts or is this normal behaviour?

Thank you in advance!

jdemaeyer commented 2 years ago

Hi @smilingchemist,

thanks for the report, I must've falsely judged a Sentry notification as "one-time error" there. Seems that restarting our worker fixed it for now, I'll look into the cause once I'm back from vacation.

Cheers, -Jakob

smilingchemist commented 2 years ago

Hi Jakob,

I fear the issue came back. I wish you a nice holiday anyway.

Frank

jdemaeyer commented 2 years ago

Digging through the Sentry issue suggests that this is based on worker processes breaking while processing MOSMIX files. Maybe it's some memory issue (although we do have plenty of free memory). I will try playing around with the number of huey workers and/or their type, although this is more wild guessing than actually knowing what's going on.

jdemaeyer commented 2 years ago

4f02502

jdemaeyer commented 1 year ago

Closing as this hasn't happened in forever. The memory footprint during parsing was greatly reduced when we refactored to use dwdparse.