search

Swirrl / ons-data-export

Temporary repo to keep track of the extraction of data between the PMD3 backed alpha for the COGS project, and the PMD4 staging server.
0 stars 0 forks source link

Product dimension #23

Closed jennet closed 4 years ago

jennet commented 4 years ago

Related to #20, many datasets use the Product dimension, which has no code list specified. Different datasets use values from various concept schemes in their data.

In PMD4, this currently means the pickers cannot display for this dimension. In order to display useful dimension pickers on cogs-staging, we have manually updated the datasets to use different dimensions that are more specific to the actual codes used in their data. However, this is a fairly time consuming process and it would be best to avoid reimporting any data that uses this dimension before we have decided the best way to proceed with the data modeling around dimensions and code lists (this is an open issue in the PMD4 technical development team)

jennet commented 4 years ago

This CSV shows the various usages of the Products dimension: https://github.com/Swirrl/ons-data-export/blob/master/data-issues/dimensions_and_data_values/components_using_product_dimension.csv

New dimensions created for cogs-staging:

     <http://gss-data.org.uk/def/dimension/bop-services> 
      <http://www.w3.org/2000/01/rdf-schema#label> "BOP Services"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/bop-services> .
    <http://gss-data.org.uk/def/dimension/hmrc-industry-groups> 
      <http://www.w3.org/2000/01/rdf-schema#label> "HMRC Industry Groups"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/hmrc-industry-groups> .
    <http://gss-data.org.uk/def/dimension/itis-services> 
      <http://www.w3.org/2000/01/rdf-schema#label> "ITIS Services"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/itis-services> .
    <http://gss-data.org.uk/def/dimension/ons-abs-trades> 
      <http://www.w3.org/2000/01/rdf-schema#label> "ONS ABS Trades"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/ons-abs-trades> .
    <http://gss-data.org.uk/def/dimension/ons-functional-category> 
      <http://www.w3.org/2000/01/rdf-schema#label> "ONS Functional Category"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/ons-functional-category> .
    <http://gss-data.org.uk/def/dimension/pink-book-services> 
      <http://www.w3.org/2000/01/rdf-schema#label> "Pink Book Services"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/pink-book-services> .
    <http://gss-data.org.uk/def/dimension/sitc-4> 
      <http://www.w3.org/2000/01/rdf-schema#label> "SITC 4"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/sitc-4> .
    <http://gss-data.org.uk/def/dimension/cord-sitc> 
      <http://www.w3.org/2000/01/rdf-schema#label> "CORD SITC"@en ;
      qb:codeList <http://gss-data.org.uk/def/concept-scheme/cord-sitc> .
 <http://gss-data.org.uk/def/dimension/cpa> 
      <http://www.w3.org/2000/01/rdf-schema#label> "Product Classification"@en ;
      qb:codeList <http://stamina-project.org/codes/cpav2008/cpa> .
jennet commented 4 years ago

Datasets affected:

jennet commented 4 years ago

Due to the ongoing discussions around PMD4 requirements for dimension / code lists, it's not known at this point in time whether any configuration changes need to be done by ONS for this dimension. I will update this issue when I have more information.

jennet commented 4 years ago

See also this log of dimension/code list tweaks that were applied to the data extraction on Thurs 23rd https://github.com/Swirrl/ons-data-export/issues/33#issuecomment-618492003

jennet commented 4 years ago

The dataset "UK trade in goods by business characteristics 2018 - data tables" was extracted as part of update #53

This dataset set the product dimension to the code list http://gss-data.org.uk/def/concept-scheme/product although the dataset uses codes from the code list http://gss-data.org.uk/def/concept-scheme/hmrc-industry-groups

I have removed the code list http://gss-data.org.uk/def/concept-scheme/product from that dimension for the time being - to avoid a filter picker with 100% greyed out options.

The work being done on PMD4 at the moment (the combination of multiple code lists together with making sparse pickers more easy to navigate/use) will definitely help with the work around the product dimension and so it's worth staying in close communications around this.

cc @ajtucker

jennet commented 4 years ago

Moved to https://github.com/Swirrl/cogs-issues/issues/12