ncihtan / data-models

Schema.org Data Models for HTAN
MIT License
14 stars 6 forks source link

Sync released data models to BQ tables #420

Open adamjtaylor opened 1 month ago

adamjtaylor commented 1 month ago

As a data manager for HTAN I would like our internal BigQuery `htan-dcc:metadata tables to include

This allows us to ensure that we can use this in queries against our submitted manifests or other information held in BigQuery

We can extend the bq-schema workflow as follows

Add running when a release is created

on:
  push:
    branches: main
    paths: 'HTAN.model.csv'
  release:
    types: [created]
  workflow_dispatch: 

Add a job to create a versioned table if the event name is release

  add-versioned-table:
    name: Add versioned schema to BQ
    runs-on: ubuntu-latest
    needs: add-to-bq
    if: github.event_name == 'release'

Then duplicate the versioned table as latest

      - name: Duplicate versioned table as latest
        shell: bash
        run: |
          VERSION=${{ github.event.release.tag_name }}
          bq cp htan-dcc:metadata.data_model_${VERSION} htan-dcc:metadata.data_model_latest
aclayton555 commented 2 weeks ago

Please add a "critical" label if expected within phase 1.0. Or a "renewal" label if this can wait.