owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
58 stars 18 forks source link

:bar_chart: Speed up GHE & GBD #2872

Closed Marigold closed 5 days ago

Marigold commented 6 days ago

What started as performance optimizations of GHE ended up as multiple performance optimizations across ETL.

This speeds up GHE & GBD by about 40% (mostly because of categorical variables). I was thinking whether this could cause issues for other datasets and whether I should increment ETL_EPOCH to test them all, but I since it only affects categorical variables, I think it should be fine.

owidbot commented 6 days ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-ghe-performance

chart-diff: ✅ No charts for review.
data-diff: ✅ No differences found ```diff Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-06-25 09:14:56 UTC Execution time: 13.73 seconds