owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
85 stars 23 forks source link

wildfires test #3570

Closed veronikasamborska1994 closed 2 days ago

owidbot commented 2 days ago
Quick links (staging server): Site Dev Site Preview Admin Wizard Docs

Login: ssh owid@staging-site-wildfires-fix

chart-diff: ✅ No charts for review.
data-diff: ✅ No differences found ```diff Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-11-19 19:58:26 UTC Execution time: 14.82 seconds

veronikasamborska1994 commented 2 days ago

hey @Marigold I am going to merge this just bc things seem to have gone wrong with the automatic update of this data - lots of it was just missing. when I ran it locally it seemed to be completely fine so I am just pushing this for now - no idea how to even replicate it tbh but since you were helping me out with setting up a regular update + might have other similar datasets I thought I'd let you know! could be because there was an issue with accessing the api at the time of the update I don't have any way of throwing an error/warning about this in the snapshot step - if you have a good solution for this that wouldn't mess with the fact that this is updated weekly let me know - otherwise I can just add an exception for when it fails tmrw!

Screenshot 2024-11-19 at 21 08 55
Marigold commented 2 days ago

Whoa, good catch, @veronikasamborska1994. I checked the Git history, and it happened a few days ago—their API returned less data than usual. I remember having some issues with their API in the past.

I've added an error to trigger if the downloaded dataframe has fewer rows than the existing snapshot. At least we'll be able to see how often it fails.