Add at least one dataset level tag to all datasets that have new metadata

[danyx23]

For the new datapages it would be extremely useful if we could guarantee that every indicator has at least one topic_tag assigned. This will ensure that we can link our topic pages with our data pages and help in showing the most relevant content in various places.

The easiest way to ensure this is to set one (or more) tag at the dataset level. This is not always appropriate but works in many cases. Tags have to match a topic tag in the database (case sensitive). The schema has been updated to include the current topic tags so you should get autocomplete in the yaml file for this field. The only non-topic that that can be used in "Uncategorized" which can be used for cases where you explicitly don't want to assign a topic (e.g. useful in cases like the WDI where every indicator needs a human or AI assigned tag and you don't have the time to do it yet).

The easiest way to assign a tag to all indicators in a dataset is to set it via the common section in the yaml file:

definitions:
  common:
    presentation:
      topic_tags:
      - Poverty

As of today we have about 100 datasets that have at least one indicator with new metadata. The list below has a checkmark to indicate if a tag has been added. Then it states the number of indicator the dataset has, the dataset name and the name is linked to the probable location of the metadata yaml file that you need to edit with the above snippet.

@lucasrodes

From Lucas

(working on #1869)

• [x] 2 - Population density (various sources, 2023.2)
• [x] 7 - Population (various sources, 2023.1)
• [x] 682 - Human Mortality Database
• [x] 168 - Life tables
• [x] 54 - Life Expectancy (period)
• [x] 3 - Life expectancy at birth
• [x] 6 - Life Expectancy: Broken limits
• [x] 33 - Survivorship percentiles (HMD, Alvarez and Vaupel; 2023)
• [x] 1 - Outsurvival statistic
• [x] 2 - Gini coefficient of lifespan inequality (HMD, UN WPP, Aburto et al.; 2023)
• [x] 37 - History of war (UCDP, 2023)
• [x] 21 - History of war (COW, 2020)
• [x] 26 - History of war (COW MID, 2020)
• [x] 12 - History of war (Gibler and Miller, 2020)
• [x] 12 - History of war (Project Mars, 2020)
• [x] 30 - History of war (PRIO v3.1, 2017)
• [x] 9 - History of war (Brecke, 1999)
• [x] 102 - History of war (UCDP, 2023)
• [x] 27 - History of war (UCDP/PRIO, 2023)
• [x] 56 - History of war (COW, 2020)
• [x] 78 - History of war (COW MID, 2020)
• [x] 34 - History of war (Gibler and Miller, 2020)
• [x] 30 - History of war (Project Mars, 2020)
• [x] 72 - History of war (PRIO v3.1, 2017)
• [x] 24 - History of war (Brecke, 1999)
• [x] 6 - Peace data (Diehl et al., 2023)
• [x] 2 - States in regions (Gleditsch)
• [x] 2 - States in regions (COW)
• [x] 2 - States in regions (ISD)
• [x] 7 - Global AIDS Update (UNAIDS, 2023)
• [x] 4 - International Completeness of Death Registration 2015-2019 (Karlinsky, 2023)
• [x] 34 - Comtrade Database
• [x] 1 - Population doubling times (various sources, 2023.1)

From Veronika

(https://github.com/owid/etl/pull/1872)

• [x] 1 - Global Plastics Outlook - Plastics use by region - projections
• [x] 13 - Global Plastics Outlook - Plastics waste by region and end-of-life fate - projections
• [x] 15 - Global Plastics Outlook - Plastics use by polymer
• [x] 16 - Global Plastics Outlook - Plastics use by application
  • [x] 20 - Global Plastics Outlook - Plastics waste in 2019 by region and application
  • [x] 20 - Global Plastics Outlook - Plastics waste in 2019 by region and polymer
• [x] 4 - Global Plastics Outlook - Greenhouse gas emissions from primary plastics
• [x] 4 - Global Plastics Outlook - Plastic leakage to aquatic environments - projections
• [x] 18 - Global Plastics Outlook - Plastics waste by region and end-of-life fate
• [x] 8 - Global Plastics Outlook - Plastic leakage to aquatic environments
• [x] 3 - Global Plastics Outlook - Plastic leakage to the environment - projections
  • [x] 2 - Production, use, and fate of all plastics ever made (Geyer et al.)
• [x] 2 - Global Corporate Investment in AI by Investment Activity, 2013ΓÇô22 (AI Index, 2023)
• [x] 2 - Use of statins for the prevention of cardiovascular disease in 41 low-income and middle-income countries.
• [x] 120 - Human Capital in the Long Run (Lee and Lee, 2016)
• [x] 2 - Global, regional and national trends in statin utilisation in high-income and low/middle-income countries, 2015-2020.
  • [x] 4 - Private Investment in AI by Focus Area (AI Index, 2023)
  • [x] 330 - Average hours of work, by education level and economic activity (ILOSTAT)
  • [x] 96 - Employment rates, by age and education level (ILOSTAT)
  • [x] 96 - Unemployment rates, by age and education level (ILOSTAT)
• [x] 68 - ESC Cardiovascular Statistics
• [x] 4 - Global atlas of medical devices

From Fiona

(https://github.com/owid/etl/pull/1872)

• [x] 2 - Historical Series of Phenological data for Cherry Tree Flowering at Kyoto City (and March Mean Temperature Reconstructions)
  • [x] 3 - Babies per woman (total fertility rate)
  • [x] 4 - Long-run child mortality (Gapminder; UN IGME)
  • [x] 10 - Leading causes of death in all ages and under-fives (IHME - GBD, 2019)

@pabloarosado

From Pablo R

• [x] 429 - Natural disasters
• [x] 90 - Global Carbon Budget (Global Carbon Project, 2023b)
• [x] 87 - Global Carbon Budget
• [x] 21 - Energy mix (Energy Institute, 2023)
• [x] 18 - Emissions-weighted Carbon Price
• [x] 18 - Global natural disasters by type
• [x] 15 - Electricity mix (EI & Ember, 2023)
• [x] 10 - Global hen inventory (Welfare Footprint Project, 2022)
• [x] 9 - Number of farmed fish killed for food worldwide
• [x] 7 - Dummy monster
• [x] 6 - Number of individual wild fish killed for food
• [x] 6 - Number of farmed decapod crustaceans killed for food (Fishcount, 2018)
• [x] 6 - Number of farmed fish killed for food (Fishcount, 2018)
• [x] 5 - UK egg statistics (Defra, 2023)
• [x] 4 - Fur banning
• [x] 4 - Fur banning
• [x] 3 - Near-surface temperature anomaly
• [x] 3 - Fossil fuel production (EI & Shift, 2023)
• [x] 2 - World Carbon Pricing Database
• [x] 2 - Net Zero Tracker
• [x] 2 - Share of cage-free eggs (Various sources, 2023)
• [x] 2 - Primary energy consumption (EI & EIA, 2023)
• [x] 1 - Emission Factor Database
• [x] 1 - Bullfighting laws
• [x] 1 - Chick culling laws
• [x] 1 - Dummy OWID dataset title

From Mojmir

• [x] 3 - Living Planet Index ΓÇô WWF (2022)
• [x] 448 - World Inequality Database (WID, 2023)
• [x] 89 - World Bank Poverty and Inequality Platform (PIP)

From Pablo A

• [x] 21 - State Capacity Dataset
• [x] 3 - Maddison Project Database
• [x] 355 - Democracy and Human rights - OWID based on Varieties of Democracy (v13) and Regimes of the World
• [x] 7 - Colonial Dates Dataset (COLDAT)
• [x] 9 - Augmented Human Development Index (AHDI)

Others

• [x] 1 - [origin.title]
• [x] 1 - Reported Smallpox Cases (WHO, 2023)
• [x] 1 - World Development Indicators - World Bank (2023.05.11)
• [x] 1 - United Nations Inter-agency Group for Child Mortality Estimation (2023)

Datasette query for the above list

The following datasette query was used to arrive at the above list: ```sql with datasets_with_counts as ( select d.name as name, count(*) as variableCount, json_group_array(distinct v.catalogPath) as paths from datasets d join variables v on v.datasetId = d.id where v.schemaVersion = 2 group by d.id order by count(*) desc ) SELECT name, variableCount, json_each.value AS path FROM datasets_with_counts t, json_each(t.paths) ```

[lucasrodes]

I've restructured the list, making it easier to work on my datasets. I'd suggest you do the same, @pabloarosado. With the ones remaining, we can either distribute them among us both or let others know. Can decide once we're finished with ours.

[pabloarosado]

I've added tags to all datasets I've worked on: https://github.com/owid/etl/pull/1870 I've realised some of the datasets listed above are still using old metadata (e.g. Energy mix). I haven't added topic tags to those.

We can just distribute of the list half and half among us, and then add the others as reviewers.

[pabloarosado]

I see that grapher steps are failing (also in my local grapher). It's related to grapher_model, something connected to the displayOrder of the topic tags. Maybe some migration needs to be done in the database. @danyx23 were you expecting this or is it a bug?

[lucasrodes]

@pabloarosado when are you getting this error?

I was getting this error when using Jinja on topic_tags (see my comment).

Could be related to this update https://github.com/owid/etl/pull/1863? @Marigold

[Marigold]

@pabloarosado sorry, my bad! Could you please rebase on top of the master?

[lucasrodes]

@pabloarosado I've re-structured the list again to clarify which datasets each of us is tackling.

I've finished my part, you can review my changes here:

• https://github.com/owid/etl/pull/1869

I've created an additional PR with the changes affecting datasets from others (no need to review this):

• https://github.com/owid/etl/pull/1872

[pabloarosado]

It's not clear to me what to do with auxiliary indicators, like population or GDP, that we have sometimes within a dataset. For example, natural disasters has those indicators. These indicators only exist because it's convenient (for us and for superuser downloading the dataset) to have population and GDP next to the per capita and per gdp variables. They're useful to do sanity checks.

• We don't want to tag them as Population or Economic Growth, because they shouldn't appear anywhere on those topics.
• We don't want to tag them as Natural Disasters, since we don't want them to appear on those searches.

So, I'll tag them as "Uncategorized". Do you agree @lucasrodes ?

[lucasrodes]

Yeah, that sounds reasonable, @pabloarosado!

[pabloarosado]

I've checked most of the datasets in the list, since they were using the old metadata (including WDI). I understand that they don't need to be tagged.

@paarriagadap among the tags in this list, which ones would you assign (being the first tag the most important one) to the indicators of:

• State Capacity Dataset.
• Colonial Dates Dataset.
• Augmented Human Development Index.

@spoonerf among the tags in this list, which ones would you assign (being the first tag the most important one) to the indicators of:

• United Nations Inter-agency Group for Child Mortality Estimation.

Thanks both!

[spoonerf]

Hey @pabloarosado!

I would go with Child & Infant Mortality for the United Nations Inter-agency Group for Child Mortality Estimation dataset.

Just be aware that the Datasette link you shared is showing only the first 101 topic tags ordered by id, but there are 128 topic tags. Maybe this link is better.

[lucasrodes]

@pabloarosado I'd use 'State Capacity' for 'State Capacity Dataset' and 'Colonial Dates Dataset'. And possible 'Human Development Index (AHDI)' for the 'Augmented Human Development Index'

[paarriagadap]

Hi! I was away last Friday. Yes, the ones described by @lucasrodes are the ones to use. It's Human Development Index (HDI), by the way.

[pabloarosado]

Hey @lucasrodes I think the two PRs are ready to be merged (assuming that there are no other new CI surprises with random steps). If everything goes well, shall we merge by the end of the day?

[pabloarosado]

This issue was completed late October, but I suppose we forgot to close it.