Missing tables in the `raw` export

[sol1105]

As discussed in #8 the raw export (dreq_raw_export.json) does not include the following tables: 'Ranking', 'Priority Level', 'Glossary', 'Table Identifiers', 'Frequency', 'Modelling Realm', 'Data Request Themes'

Since the release export (dreq_release_export.json) is supposed to be a mapped subset of the raw export, the raw export should include this tables. Until it does, a consolidation process taking a raw export to yield a 1:1 copy of a release export is not possible.

In the following the mapping conducted by Beth in Airtable:

V1.0beta table name	Source table base	Source table name
Opportunity	Data Request Opportunities	Opportunity
Variable Group	Data Request Opportunities	Variable Group
Variables	Data Request Variables	Variables
Experiment Group	Data Request Opportunities	Experiment Group
Experiments	Data Request Opportunities	Experiments
Physical Parameters	Data Request Physical Parameters	Physical Parameters
MIPs	Data Request Opportunities	MIPs
Data Request Themes	Data Request Opportunities	Data Request Themes
Priority Level	Data Request Opportunities	Priority Level
Docs for Opportunities	Data Request Opportunities	Docs for Opportunities
Time slice	Data Request Opportunities	Time slice
Opportunity/variable group comments	Data Request Opportunities	Comments
Glossary	Data Request Opportunities	Glossary
Modelling realm	Data Request Variables	Modelling realm
Frequency	Data Request Variables	Frequency
Table identifiers	Data Request Variables	Table identifiers
Spatial shape	Data Request Variables	Spatial shape
Temporal shape	Data Request Variables	Temporal shape
Structure	Data Request Variables	Structure
Cell methods	Data Request Variables	Cell methods
Cell measures	Data Request Variables	Cell measures
Coordinates and dimensions	Data Request Variables	Coordinates and dimensions
Ranking	Data Request Variables	Ranking
Variable comments	Data Request Variables	Comments
ESM-BCV1.3	Data Request Variables	ESM-BCV1.3
CF Standard names	Data Request Physical Parameters	CF Standard names
Physical parameter comments	Data Request Physical Parameters	Comments

[CMIP-IPO]

We added a table to the v1.0beta release. An updated mapping is below (the changes are to the Frequency table(s)).

V1.0beta table name	Source table base	Source table name
Opportunity	Data Request Opportunities	Opportunity
Variable Group	Data Request Opportunities	Variable Group
Variables	Data Request Variables	Variables
Experiment Group	Data Request Opportunities	Experiment Group
Experiments	Data Request Opportunities	Experiments
Physical Parameters	Data Request Physical Parameters	Physical Parameters
MIPs	Data Request Opportunities	MIPs
Data Request Themes	Data Request Opportunities	Data Request Themes
Priority Level	Data Request Opportunities	Priority Level
Docs for Opportunities	Data Request Opportunities	Docs for Opportunities
Time slice	Data Request Opportunities	Time slice
Opportunity/variable group comments	Data Request Opportunities	Comments
Glossary	Data Request Opportunities	Glossary
Modelling realm	Data Request Variables	Modelling realm
CMIP6 Frequency (legacy)	Data Request Variables	CMIP6 Frequency (legacy)
CMIP7 Frequency	Data Request Variables	CMIP7 Frequency
Table identifiers	Data Request Variables	Table identifiers
Spatial shape	Data Request Variables	Spatial shape
Temporal shape	Data Request Variables	Temporal shape
Structure	Data Request Variables	Structure
Cell methods	Data Request Variables	Cell methods
Cell measures	Data Request Variables	Cell measures
Coordinates and dimensions	Data Request Variables	Coordinates and dimensions
Ranking	Data Request Variables	Ranking
Variable comments	Data Request Variables	Comments
ESM-BCV1.3	Data Request Variables	ESM-BCV1.3
CF Standard names	Data Request Physical Parameters	CF Standard names
Physical parameter comments	Data Request Physical Parameters	Comments

Discussed in the TISG meeting on 24th October that the missing tables are due to issues using the public bases to do the export. We decided to use the full backend Airtable for exports. This will resolve the missing table issue.

[CMIP-IPO]

Filters applied in the consolidation of the bases: In the Opportunities table: Status=="Under review" OR Status=="Accepted"

In the Variable Groups table: (Final Opportunity selection is not empty) AND (Opportunity Status =="Under review" OR Opportunity Status=="Accepted")

In the Experiment Groups table: (Status (from Opportunity)==New OR Status (from Opportunity)==Under Review OR Status (from Opportunity)==Accepted) AND (Status!=Junk)

In the Variables table: (CMIP7 Variable Groups is not empty) AND (Opportunity Status (from CMIP7 Variable Groups)==Under review OR Opportunity Status (from CMIP7 Variable Groups)==Accepted)

[sol1105]

@CMIP-IPO Thank you for listing the filters. In the latest export there are however some of the fields missing, eg. Status (from Opportunity) does currently not exist in the Experiment Groups table.

We decided to use the full backend Airtable for exports. This will resolve the missing table issue.

Will this also be resolved by no longer using the public bases for exports?

[sol1105]

@CMIP-IPO As discussed in todays TISG meeting, the list of missing fields:

• Opportunity Base
  • Variable Group
  • Opportunity Status (There is however Status (from Final Opportunity selection) if that should be used)
  • Experiment Group
  • Status (from Opportunity)
• Variables Base
  • Variable
  • Opportunity Status (from CMIP7 Variable Groups)

[CMIP-IPO]

@sol1105

• Opportunity Base
  • Variable Group
  • Opportunity Status (There is however Status (from Final Opportunity selection) if that should be used)
  • You are right, those two fields are the same. I've changed the name in the public base to Opportunity Status
  • Experiment Group
  • Status (from Opportunity) ADDED
• Variables Base
  • Variable
  • Opportunity Status (from CMIP7 Variable Groups) ADDED