Implement the ingestion of the Performance hub metadata

[teeceeas]

As discussed in the spike an automated / API solution may be possible in future but for now we will ingest in a similar way to ESDAs - writing a script to scan a CSV and push catalogue entries via the DataHub API.

Ingestion

• Metrics can be assigned to a new "database" container called "HMPPS Performance Hub"
• Title - I suggest using ShortTitle. "Report title" is fairly legacy so can be ignored (in most cases the short title is the same).
• Description - there aren't more detailed descriptions. Suggest constructing one from "Performance Hub metric $shortTitle ($metricRef). Sourced from $dataSource.". E.g. "Performance Hub metric: Approved premises RoSH level (AP01 MI001). Sourced from nDelius extract." (Some users will know metric ref - e.g. probation performance is published with this, so being searchable is good). There is an additional block of text below to be added to all descriptions.
• Subject area - see below
• Data owner - leave empty for now (see below)
• Request access - leave empty for now
• Ask a question - leave empty for now
• There aren't any instances now, but just in case, don't ingest any records where Active = false

Subject areas

• If EntityType = Prison, assign it to the prison subject area
• Else probation (that includes, BASS, EM and approved premises)

Description

In addition to the above, append the following three sentences:

No owner is listed as this data is undergoing a review of ownership.

Data quality: the data is checked to be good enough to support the outcomes it is being used for. Data values should be right, but there are other factors that help ensure data meets the needs of its users.

Please remember that this data is for INTERNAL USE ONLY and not to be shared outside the organisation.

Data owners

Assigning data ownership has been de-scoped following a conversation with D&A (the previously suggested ownership is saved below in a comment).

[MatMoore]

The spike ticket says the result will likely look like a chart but I'll need to check whether we can actually link to the chart like we did with with Justice Data.

Both charts and datasets can be specialised with SubType. Setting this to a value like "PerformanceMetric" preserves the option of handling metrics differently to other kinds of charts.

[jemnery]

This is what the ownership logic was - we're not implementing this

• If a metric has been assigned to the probation subject area, add "hubprobation@justice.gov.uk" as the owner
• For prison subject area metrics:
  • If DataSource is "Finds Upload", "Incidents Upload" "Incidents Upload (IRS)" or "Hub upload (assaults & referrals)" assign HubNationalOperation@justice.gov.uk as the owner
  • If DataSource is "Alere Tox lab data", assign HubMDT@justice.gov.uk as the owner ("MDT" is mandatory drug testing in prisons)
  • If DataSource is "Programmes return" or "HII programme uploads" OR the MetricRef starts with "CUP", assign HubProgrammesMinorities@justice.gov.uk as the owner
  • Otherwise assign hubprisons@justice.gov.uk as the owner

[MatMoore]

Work in progress, but have something ingested https://dev.find-moj-data.service.justice.gov.uk/details/dashboard/urn:li:dashboard:(performance-hub,performance-hub)

We're going to need to adjust the templates in Find MoJ data

• Charts have hardcoded links to Justice data - this should be based on platform
• There are two many metrics to display on one page - we really need some kind of pagination or search

The subject area is missing for dashboard but since it covers multiple domains I'm going to leave this blank. This is the same for Justice Data as well.

[MatMoore]

Chart/dashboard is not actually a very good description of what these metrics are, so I'm going to change it to Dataset/Container, and introduce a subtype for Metrics.

We can then hide the schema part from the template.