Transfer dataset descriptions from trade site

[Robsteranium]

The trade site had rich dataset descriptions (example). These haven't made it into the new database yet (example). We ought to transfer them.

We might also want to transfer the structured metadata. It might make sense to review these properties (as we're essentially deciding to provide them for all datasets, not just trade ones). The beta/ staging PMD deployments will need to be reconfigured if we decide to adopt them.

[Robsteranium]

The strengths/ weakness might actually be a nice summary to surface on the structured search prototype.

[JasonHowell]

The Trade portal site with the additional data descriptions need to be assessed and migrated over where appropriate.

Need someone to go through and validate the appropriateness of the data description and make a call on whether to transfer the description across to the Beta site.

Example: https://trade.gss-data.org.uk/

https://trade.gss-data.org.uk/dataset/about?uri=http%3A%2F%2Fgss-data.org.uk%2Fdata%2Fgss_data%2Ftrade%2Fons_pinkbook-catalog-entry

The About tab > description.

[ajtucker]

These additional descriptions came from a number of templated documents supplied to us by the trade consortium, so we could follow the chain back and ask for a review to ensure the descriptions are still valid.

I've added an issue to help get the extra metadata descriptions into our pipelines GSS-Cogs/csvcubed#505. We'd be able to programatically add the extra descriptions as part of a Jupyter notebook.

Ideally, these extra descriptions should be in a separate structured doc we can track in Git. For now, I'd suggest using info.json and follow up GSS-Cogs/gss-utils#38.

[Robsteranium]

Note that we've already seen evidence that the imported descriptions don't match the cube contents - e.g. the ABS Importers and Exporters description promises a breakdown by SIC07, this is available in the ABS itself but not the Importers and Exporters dataset.

I'd suggest we ask the BA team to review them instead (as they'll have a better understanding of the context).

re: the following the chain: I don't have access to the sharepoint (?) system where DIT keep the original docs but we have a 14 month old snapshot in the ons-data-export repo - just ask if you need access/ want me to pop a copy here.

Any thoughts on the structured metadata @ajtucker @JasonHowell?

[JasonHowell]

Have submitted a ticket onto the Business Analytsis Trello board:

https://trello.com/c/fTqKk3qP/95-trade-descriptions

[BillSwirrl]

A subset of the datasets on https://trade.gss-data.org.uk/ have relatively rich descriptions and other descriptive metadata items. The source of this was a set of Word files put together by DIT and held internally on their sharepoint system. Darren obtained copies of (some of) them for us. Copies of the Word files are in the zip attached to this comment. DIT-descriptions.zip

These are the datasets with this detailed metadata:

03 Trade in services, The Pink Book

Annual Business Survey exporters and importers

Foreign direct investment involving UK companies

International exports of services from subnational areas of the UK

Overseas Trade Statistics (CN8)

Trade in goods: all countries, seasonally adjusted

UK Regional Trade Statistics (RTS)

UK trade in goods by classification of product by activity time series

UK trade in goods by industry, country and commodity, imports and exports

UK trade in services: service type by partner country, non-seasonally adjusted

UK trade time series

[JasonHowell]

This might be out of date as publications have incremented and moved on?