owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
86 stars 23 forks source link

:hammer: Remove archived/ folder with steps and --archive flag #3553

Open Marigold opened 1 week ago

owidbot commented 1 week ago
Quick links (staging server): Site Dev Site Preview Admin Wizard Docs

Login: ssh owid@staging-site-removed-archived-steps

chart-diff: ✅ No charts for review.
data-diff: ❌ Found differences ```diff = Dataset garden/education/2023-08-14/oecd_education = Table oecd_education ~ Dim country + + New values: 125 / 7000 (1.79%) year country 2023 Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius - - Removed values: 363 / 7000 (5.19%) year country 2009 Cayman Islands 2012 Curacao 2015 Netherlands Antilles 2007 South Sudan 2021 South Sudan ~ Dim year + + New values: 125 / 7000 (1.79%) country year Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 - - Removed values: 363 / 7000 (5.19%) country year Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 ~ Column illiterate (changed metadata, new data, changed data) + + description_short: Percentage of people aged 15 and above who can read and write a short, simple statement on their everyday + + life. + + url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics - - version_producer: '2023' - - url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2023-07-26' ? ^ ^^^^ + + date_accessed: '2024-11-04' ? ^^^^ ^ - - date_published: '2023' ? ^ + + date_published: '2024-10-01' ? ^^^^^^^ + + New values: 125 / 7000 (1.79%) country year illiterate Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 - - Removed values: 363 / 7000 (5.19%) country year illiterate Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 ~ Changed values: 815 / 7000 (11.64%) country year illiterate - illiterate + Arab World (WB) 2022 24.828468 24.181541 Bangladesh 2021 24.0 El Salvador 2009 15.89711 16.0 Pakistan 2010 44.624809 45.0 South Asia (WB) 2019 28.057213 28.033974 ~ Column literacy (changed metadata, new data, changed data) + + description_short: Percentage of people aged 15 and above who can read and write a short, simple statement on their everyday + + life. + + url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics - - version_producer: '2023' - - url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2023-07-26' ? ^ ^^^^ + + date_accessed: '2024-11-04' ? ^^^^ ^ - - date_published: '2023' ? ^ + + date_published: '2024-10-01' ? ^^^^^^^ + + New values: 125 / 7000 (1.79%) country year literacy Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 - - Removed values: 363 / 7000 (5.19%) country year literacy Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 ~ Changed values: 811 / 7000 (11.59%) country year literacy - literacy + Arab World (WB) 2022 75.171532 75.818459 Bangladesh 2021 76.0 El Salvador 2011 85.493988 85.0 Pakistan 2011 54.738022 55.0 South Asia (WB) 2018 72.081512 72.192062 ~ Column no_formal_education (changed metadata, new data, changed data) + + description_short: Percentage of the population aged 15 and older with no formal education. + + url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics - - version_producer: '2023' - - url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2023-07-26' ? ^ ^^^^ + + date_accessed: '2024-11-04' ? ^^^^ ^ - - date_published: '2023' ? ^ + + date_published: '2024-10-01' ? ^^^^^^^ + + New values: 125 / 7000 (1.79%) country year no_formal_education Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 - - Removed values: 363 / 7000 (5.19%) country year no_formal_education Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 ~ Changed values: 1688 / 7000 (24.11%) country year no_formal_education - no_formal_education + Aruba 2035 3.0 0.03 Guinea 2040 39.0 0.39 India 2100 2.0 0.02 Luxembourg 2045 2.0 0.02 North Macedonia 2020 2.0 0.02 ~ Column population_with_basic_education (changed metadata, new data, changed data) + + description_short: Percentage of the population aged 15 and older with no formal education. + + url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics - - version_producer: '2023' - - url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2023-07-26' ? ^ ^^^^ + + date_accessed: '2024-11-04' ? ^^^^ ^ - - date_published: '2023' ? ^ + + date_published: '2024-10-01' ? ^^^^^^^ + + New values: 125 / 7000 (1.79%) country year population_with_basic_education Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 - - Removed values: 363 / 7000 (5.19%) country year population_with_basic_education Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 ~ Changed values: 1688 / 7000 (24.11%) country year population_with_basic_education - population_with_basic_education + Aruba 2035 97.0 99.970001 Guinea 2040 61.0 99.610001 India 2100 98.0 99.980003 Luxembourg 2045 98.0 99.980003 North Macedonia 2020 98.0 99.980003 = Dataset garden/education/2023-12-15/wittgenstein_center = Table wittgenstein_center_data ~ Dim country - - Removed values: 285 / 7022 (4.06%) year country 2012 Faroe Islands 2019 Guam 2007 North Korea 2017 North Korea 2008 Puerto Rico ~ Dim year - - Removed values: 285 / 7022 (4.06%) country year Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Column no_formal_education (changed metadata, changed data) + + url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics - - version_producer: '2023' - - url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2023-07-26' ? ^ ^^^^ + + date_accessed: '2024-11-04' ? ^^^^ ^ - - date_published: '2023' ? ^ + + date_published: '2024-10-01' ? ^^^^^^^ - - Removed values: 285 / 7022 (4.06%) country year no_formal_education Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Changed values: 281 / 7022 (4.00%) country year no_formal_education - no_formal_education + Azerbaijan 2010 1.0 0.01 Cote d'Ivoire 2010 46.0 0.46 Ethiopia 2015 43.0 0.43 Iran 2015 15.000001 0.15 Sao Tome and Principe 2010 9.0 0.09 ~ Column population_with_basic_education (changed metadata, changed data) + + url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics - - version_producer: '2023' - - url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2023-07-26' ? ^ ^^^^ + + date_accessed: '2024-11-04' ? ^^^^ ^ - - date_published: '2023' ? ^ + + date_published: '2024-10-01' ? ^^^^^^^ - - Removed values: 285 / 7022 (4.06%) country year population_with_basic_education Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Changed values: 281 / 7022 (4.00%) country year population_with_basic_education - population_with_basic_education + Azerbaijan 2010 99.0 99.989998 Cote d'Ivoire 2010 54.0 99.540001 Ethiopia 2015 57.0 99.57 Iran 2015 85.0 99.849998 Sao Tome and Principe 2010 91.0 99.910004 + Dataset garden/wb/2024-11-04/edstats + + Table edstats + + Column expected_years_of_school + + Column harmonized_test_scores + + Column learning_adjusted_years_of_school + + Column piaac__mean_adult_literacy_proficiency__total + + Column piaac__mean_adult_numeracy_proficiency__total + + Column piaac__mean_young_adult_numeracy_proficiency__total + + Column pisa__female_15_year_olds_by_mathematics_proficiency_level__pct__level_6 + + Column pisa__male_15_year_olds_by_mathematics_proficiency_level__pct__level_6 + + Column annual_statutory_teacher_salaries_in_public_institutions_in_usd__primary__10_years_of_experience + + Column wittgenstein_projection__percentage_of_the_population_age_15plus_by_highest_level_of_educational_attainment__no_education__total + + Column saber__teachers__policy_goal_7__supporting_teachers_to_improve_instruction + + Column literacy_rate__adult_total__pct_of_people_ages_15_and_above + + Column school_enrollment__preprimary__pct_gross + + Column trained_teachers_in_primary_education__pct_of_total_teachers + + Column trained_teachers_in_secondary_education__pct_of_total_teachers + + Column government_expenditure_on_education__total__pct_of_gdp + + Column literacy_rate__population_25_64_years__both_sexes__pct + + Column total_net_enrolment_rate__primary__gender_parity_index__gpi + + Column total_net_enrolment_rate__lower_secondary__gender_parity_index__gpi + + Column out_of_school_children__adolescents_and_youth_of_primary_and_secondary_school_age__female__number + + Column out_of_school_children__adolescents_and_youth_of_primary_and_secondary_school_age__male__number + + Column school_life_expectancy__primary__gender_parity_index__gpi + + Column normalized_hci + + Column combined_literacy + + Column combined_expenditure Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-11-21 09:52:38 UTC Execution time: 17.36 seconds