Data catalogue

This is the root repository for the MOJ data catalogue, and contains the deployment workflows for the Datahub backend.

Component repositories

• find-moj-data - a view of the catalogue with GOV.UK styles
• data-catalogue-metadata (internal) contains JSON files used to populate the catalogue in development

Environments

• find-moj-data (dev)

• find-moj-data (test)

• find-moj-data (preprod)

• DataHub (dev)

• DataHub (test)

• DataHub (preprod)

Managing database secrets

The k8s-rds-secrets-to-github-secrets.sh script can be used from a local device to populate this repo's environment secrets with the database secrets for the datahub environments deployed in cloud platform.

It can be called as

bash k8s-rds-secrets-to-github-secrets <namespace root> <CP environment name>

for example:

bash k8s-rds-secrets-to-github-secrets.sh data-platform-datahub-catalogue dev

Administration of Datahub via command line

First time setup

Run datahub init and provide the following credentials

• server: https://datahub-catalogue-dev.apps.live.cloud-platform.service.justice.gov.uk
• token: <generate PAT via the UI>

You may also need to set the environment variable export DATAHUB_GMS_URL="https://datahub-catalogue-dev.apps.live.cloud-platform.service.justice.gov.uk/api/gms"

Import metadata into a Datahub lite

Datahub lite is a developer interface for local debugging.

lite_sink.yaml:

pipeline_name: datahub_source_1
datahub_api:
  server: "https://datahub-catalogue-dev.apps.live.cloud-platform.service.justice.gov.uk/api/gms"
  token: "xxxxx"
source:
  type: datahub
  config:
    include_all_versions: false
    pull_from_datahub_api: true
sink:
  type: datahub-lite

datahub ingest -c lite_sink.yaml

datahub lite ls