owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
85 stars 23 forks source link

:bar_chart: include un desa migrant stock flows in grapher #3581

Open antea04 opened 1 day ago

antea04 commented 1 day ago

adding un desa migrant stock [destination and origin].

Simon needs this data for his upcoming article on migration (distances), also I think it's nice to have this data for any DIs etc - even if we do not change the Migration Flows explorer to being etl based immediately.

owidbot commented 1 day ago
Quick links (staging server): Site Dev Site Preview Admin Wizard Docs

Login: ssh owid@staging-site-undesa-flows

chart-diff: ✅
  • 1/1 reviewed charts
  • Modified: 0/0
  • New: 1/1
  • Rejected: 0
data-diff: ❌ Found differences ```diff = Dataset garden/education/2023-08-14/oecd_education = Table oecd_education ~ Dim country + + New values: 363 / 7238 (5.02%) year country 2009 Cayman Islands 2012 Curacao 2015 Netherlands Antilles 2007 South Sudan 2021 South Sudan - - Removed values: 125 / 7238 (1.73%) year country 2023 Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius ~ Dim year + + New values: 363 / 7238 (5.02%) country year Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 - - Removed values: 125 / 7238 (1.73%) country year Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Column illiterate (changed metadata, new data, changed data) - - description_short: Percentage of people aged 15 and above who can read and write a short, simple statement on their everyday - - life. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year illiterate Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 - - Removed values: 125 / 7238 (1.73%) country year illiterate Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 815 / 7238 (11.26%) country year illiterate - illiterate + Arab World (WB) 2022 24.181541 24.828468 Bangladesh 2021 24.0 El Salvador 2009 16.0 15.89711 Pakistan 2010 45.0 44.624809 South Asia (WB) 2019 28.033974 28.057213 ~ Column literacy (changed metadata, new data, changed data) - - description_short: Percentage of people aged 15 and above who can read and write a short, simple statement on their everyday - - life. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year literacy Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 - - Removed values: 125 / 7238 (1.73%) country year literacy Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 811 / 7238 (11.20%) country year literacy - literacy + Arab World (WB) 2022 75.818459 75.171532 Bangladesh 2021 76.0 El Salvador 2011 85.0 85.493988 Pakistan 2011 55.0 54.738022 South Asia (WB) 2018 72.192062 72.081512 ~ Column no_formal_education (changed metadata, new data, changed data) - - description_short: Percentage of the population aged 15 and older with no formal education. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year no_formal_education Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 - - Removed values: 125 / 7238 (1.73%) country year no_formal_education Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 1688 / 7238 (23.32%) country year no_formal_education - no_formal_education + Aruba 2035 0.03 3.0 Guinea 2040 0.39 39.0 India 2100 0.02 2.0 Luxembourg 2045 0.02 2.0 North Macedonia 2020 0.02 2.0 ~ Column population_with_basic_education (changed metadata, new data, changed data) - - description_short: Percentage of the population aged 15 and older with no formal education. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year population_with_basic_education Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 - - Removed values: 125 / 7238 (1.73%) country year population_with_basic_education Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 1688 / 7238 (23.32%) country year population_with_basic_education - population_with_basic_education + Aruba 2035 99.970001 97.0 Guinea 2040 99.610001 61.0 India 2100 99.980003 98.0 Luxembourg 2045 99.980003 98.0 North Macedonia 2020 99.980003 98.0 = Dataset garden/education/2023-12-15/wittgenstein_center = Table wittgenstein_center_data ~ Dim country + + New values: 285 / 7307 (3.90%) year country 2012 Faroe Islands 2019 Guam 2007 North Korea 2017 North Korea 2008 Puerto Rico ~ Dim year + + New values: 285 / 7307 (3.90%) country year Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Column no_formal_education (changed metadata, new data, changed data) - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 285 / 7307 (3.90%) country year no_formal_education Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Changed values: 281 / 7307 (3.85%) country year no_formal_education - no_formal_education + Azerbaijan 2010 0.01 1.0 Cote d'Ivoire 2010 0.46 46.0 Ethiopia 2015 0.43 43.0 Iran 2015 0.15 15.000001 Sao Tome and Principe 2010 0.09 9.0 ~ Column population_with_basic_education (changed metadata, new data, changed data) - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 285 / 7307 (3.90%) country year population_with_basic_education Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Changed values: 281 / 7307 (3.85%) country year population_with_basic_education - population_with_basic_education + Azerbaijan 2010 99.989998 99.0 Cote d'Ivoire 2010 99.540001 54.0 Ethiopia 2015 99.57 57.0 Iran 2015 99.849998 85.0 Sao Tome and Principe 2010 99.910004 91.0 = Dataset garden/health/latest/global_health_mpox = Table global_health_mpox ~ Dim country + + New values: 1 / 137 (0.73%) date country 2024-11-13 Democratic Republic of Congo ~ Dim date + + New values: 1 / 137 (0.73%) country date Democratic Republic of Congo 2024-11-13 ~ Column reported_cases (new data, changed data) + + New values: 1 / 137 (0.73%) country date reported_cases Democratic Republic of Congo 2024-11-13 2616 ~ Changed values: 1 / 137 (0.73%) country date reported_cases - reported_cases + Democratic Republic of Congo 2024-11-03 475 1781 ~ Column suspected_cases_cumulative (new data, changed data) + + New values: 1 / 137 (0.73%) country date suspected_cases_cumulative Democratic Republic of Congo 2024-11-13 46440 ~ Changed values: 1 / 137 (0.73%) country date suspected_cases_cumulative - suspected_cases_cumulative + Democratic Republic of Congo 2024-11-03 42518 43824 = Dataset garden/who/2024-09-09/flu_test = Table flu_test ~ Dim country - - Removed values: 4 / 72198 (0.01%) date country 2024-11-11 Cameroon 2024-11-11 Canada 2024-11-11 Malaysia 2024-11-11 Singapore ~ Dim date - - Removed values: 4 / 72198 (0.01%) country date Cameroon 2024-11-11 Canada 2024-11-11 Malaysia 2024-11-11 Singapore 2024-11-11 ~ Column denomcombined (changed data) - - Removed values: 4 / 72198 (0.01%) country date denomcombined Cameroon 2024-11-11 61 Canada 2024-11-11 24586 Malaysia 2024-11-11 931 Singapore 2024-11-11 130 ~ Changed values: 7 / 72198 (0.01%) country date denomcombined - denomcombined + Australia 2024-11-04 3320 3316 Australia 2024-11-11 3037 2995 Canada 2024-10-07 24938 24937 Canada 2024-10-28 25686 25688 Malaysia 2024-09-23 924 923 ~ Column pcnt_poscombined (changed data) - - Removed values: 4 / 72198 (0.01%) country date pcnt_poscombined Cameroon 2024-11-11 29.508196 Canada 2024-11-11 0.748393 Malaysia 2024-11-11 12.352309 Singapore 2024-11-11 16.923077 ~ Changed values: 8 / 72198 (0.01%) country date pcnt_poscombined - pcnt_poscombined + Australia 2024-11-11 2.304906 0.934891 Canada 2024-10-07 0.296736 0.296748 Canada 2024-10-28 0.657946 0.657895 Malaysia 2024-09-23 20.129869 20.15168 Malaysia 2024-11-04 16.438356 16.332983 = Dataset garden/who/latest/monkeypox = Table monkeypox ~ Dim country + + New values: 1 / 97705 (0.00%) date country 2024-11-13 Democratic Republic of Congo ~ Dim date + + New values: 1 / 97705 (0.00%) country date Democratic Republic of Congo 2024-11-13 ~ Column annotation (new data) + + New values: 1 / 97705 (0.00%) country date annotation Democratic Republic of Congo 2024-11-13 Includes only confirmed deaths as reported by WHO ~ Column iso_code (new data) + + New values: 1 / 97705 (0.00%) country date iso_code Democratic Republic of Congo 2024-11-13 NaN ~ Column new_cases (new data) + + New values: 1 / 97705 (0.00%) country date new_cases Democratic Republic of Congo 2024-11-13 ~ Column new_cases_per_million (new data) + + New values: 1 / 97705 (0.00%) country date new_cases_per_million Democratic Republic of Congo 2024-11-13 ~ Column new_cases_smoothed (new data) + + New values: 1 / 97705 (0.00%) country date new_cases_smoothed Democratic Republic of Congo 2024-11-13 ~ Column new_cases_smoothed_per_million (new data) + + New values: 1 / 97705 (0.00%) country date new_cases_smoothed_per_million Democratic Republic of Congo 2024-11-13 ~ Column new_deaths (new data) + + New values: 1 / 97705 (0.00%) country date new_deaths Democratic Republic of Congo 2024-11-13 ~ Column new_deaths_per_million (new data) + + New values: 1 / 97705 (0.00%) country date new_deaths_per_million Democratic Republic of Congo 2024-11-13 ~ Column new_deaths_smoothed (new data) + + New values: 1 / 97705 (0.00%) country date new_deaths_smoothed Democratic Republic of Congo 2024-11-13 ~ Column new_deaths_smoothed_per_million (new data) + + New values: 1 / 97705 (0.00%) country date new_deaths_smoothed_per_million Democratic Republic of Congo 2024-11-13 ~ Column suspected_cases_cumulative (new data, changed data) + + New values: 1 / 97705 (0.00%) country date suspected_cases_cumulative Democratic Republic of Congo 2024-11-13 46440 ~ Changed values: 1 / 97705 (0.00%) country date suspected_cases_cumulative - suspected_cases_cumulative + Democratic Republic of Congo 2024-11-03 42518 43824 ~ Column total_cases (new data) + + New values: 1 / 97705 (0.00%) country date total_cases Democratic Republic of Congo 2024-11-13 ~ Column total_cases_per_million (new data) + + New values: 1 / 97705 (0.00%) country date total_cases_per_million Democratic Republic of Congo 2024-11-13 ~ Column total_deaths (new data) + + New values: 1 / 97705 (0.00%) country date total_deaths Democratic Republic of Congo 2024-11-13 ~ Column total_deaths_per_million (new data) + + New values: 1 / 97705 (0.00%) country date total_deaths_per_million Democratic Republic of Congo 2024-11-13 Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-11-21 16:05:25 UTC Execution time: 4.54 seconds