Please visit the Great Expectations Documentation for more information on the deployment pattern best suited to your environment.
This Action allows you to validate and profile your data with Great Expectations. From the Great Expectations docs:
Great Expectations is a leading tool for validating, documenting, and profiling your data to maintain quality and improve communication between teams.
This Action provides the following features:
great_expectations/
subdirectory), with which you have created at least one Expectation Suite and Checkpoint for the data you want to secure. See the Great Expectations Getting started tutorial for instructions on the setup.great_expectations.yml
references an environment variable DB_HOST
, this is what you need to configure in GitHub secrets.In order to configure the GitHub action for your repository, add the following code snippet to your GitHub workflows file. The file should be located under my_repo_name/.github/my_workflow.yml
.
This example triggers Great Expectations to run every time a GitHub pull request is opened, reopened, or a push is made to a pull request. Furthermore, if a Checkpoint fails, a comment with a link to the Data Docs hosted on Netlify is provided.
Note: This example will not work on pull requests from forks. This is to protect repositories from malicious actors. To trigger a GitHub Action with sufficient permissions to comment on a pull request from a fork, you must trigger the Action via another event, such as a comment or a label. This is demonstrated in Example 2 below.
# Automatically Runs Great Expectation Checkpoints on every push to a PR, and provides links to hosted Data Docs if there an error.
name: PR Push
on: pull_request
env: # Credentials to your development database (this is illustrative, can be another data source)
DB_HOST: ${{ secrets.DB_HOST }}
DB_PASS: ${{ secrets.DB_PASS }}
DB_USER: ${{ secrets.DB_USER }}
jobs:
great_expectations_validation:
runs-on: ubuntu-latest
steps:
# Clone the contents of the repository
- name: Copy Repository contents
uses: actions/checkout@main
# Execute your data pipeline on development infrastructure. This is a simplified example where
# we run a local SQL file against a remote Postgres development database as the "pipeline". We
# then test the materialized data from the pipeline with this Action.
# It is up to you to configure how you kick off your pipeline in a workflow!
- name: Run SQL "pipeline"
run: |
PGPASSWORD=${DB_PASS} psql -h $DB_HOST -d demo -U $DB_USER -f location_frequency.sql
# Run Great Expectations and deploy Data Docs to Netlify
# In this example, we have configured a Checkpoint called "locations.rds.chk".
- name: Run Great Expectation Checkpoints
id: ge
# Use @v0.x instead of @main to pin to a specific version, e.g. @v0.2
uses: great-expectations/great_expectations_action@main
with:
CHECKPOINTS: "locations.rds.chk" # This can be a comma-separated list of Checkpoints
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
# Comment on PR with link to deployed Data Docs if there is a failed Checkpoint, otherwise don't comment.
- name: Comment on PR
if: ${{ always() }}
uses: actions/github-script@v2
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
if (process.env.FAILURE_FLAG == 1 ) {
msg = `Failed Great Expectations Checkpoint(s) \`${process.env.FAILED_CHECKPOINTS}\` detected for: ${process.env.SHA}. Corresponding Data Docs have been generated and can be viewed [here](${process.env.URL}).`;
console.log(`Message to be emitted: ${msg}`);
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: msg
});
}
env:
URL: "${{ steps.ge.outputs.netlify_docs_url }}"
FAILED_CHECKPOINTS: ${{ steps.ge.outputs.failing_checkpoints }}
SHA: ${{ github.sha }}
FAILURE_FLAG: ${{ steps.ge.outputs.checkpoint_failure_flag }}
In order to configure the GitHub action for your repository, add the following code snippet to your GitHub workflows file. The file should be located under my_repo_name/.github/my_workflow.yml
.
The below example checks pull request comments for the presence of a special command: /data-docs
. If this command is present, the following steps occur:
# Allows repo owners to view Data Docs hosted on Netlify for a PR with the command "/data-docs" as a comment in a PR.
name: PR Comment
on: [issue_comment]
jobs:
demo-pr:
# Check whether a comment with the command '/data-docs' is made on a pull request (not an issue).
if: |
(github.event.issue.pull_request != null) &&
contains(github.event.comment.body, '/data-docs')
runs-on: ubuntu-latest
steps:
# Get the HEAD SHA of the pull request that has been commented on.
- name: Fetch context about the PR that has been commented on
id: chatops
uses: actions/github-script@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
// Get the branch name
github.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.issue.number
}).then( (pr) => {
// Get latest SHA of current branch
var SHA = pr.data.head.sha
console.log(`::set-output name=SHA::${SHA}`)
})
# Clone the contents of the repository at the SHA fetched in the previous step
- name: Copy the PR's branch repository contents
uses: actions/checkout@main
with:
ref: ${{ steps.chatops.outputs.SHA }}
- name: Run data pipeline on dev server
run: # <Put your code here> run your data pipeline in development so you can test the results with Great Expectations
# Run Great Expectation Checkpoints and deploy Data Docs to Netlify
- name: Run Great Expectation Checkpoints
id: ge
# Use @v0.x instead of @main to pin to a specific version, e.g. @v0.2
uses: great-expectations/great_expectations_action@main
with:
CHECKPOINTS: ${{ matrix.checkpoints }}
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
# Comment on PR with link to deployed Data Docs on Netlify. In this example, we comment
# on the PR with a message for both failed and successful checks.
- name: Comment on PR
if: ${{ always() }}
uses: actions/github-script@v2
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
if (process.env.FAILURE_FLAG == 1 ) {
msg = `Failed Great Expectations Checkpoint(s) \`${process.env.FAILED_CHECKPOINTS}\` detected for: ${process.env.SHA}. Corresponding Data Docs have been generated and can be viewed [here](${process.env.URL}).`;
} else {
msg = `All Checkpoints for: ${process.env.SHA} have passed. Corresponding Data Docs have been generated and can be viewed [here](${process.env.URL}).`;
}
console.log(`Message to be emitted: ${msg}`);
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: msg
})
env:
URL: "${{ steps.ge.outputs.netlify_docs_url }}"
FAILED_CHECKPOINTS: ${{ steps.ge.outputs.failing_checkpoints }}
SHA: ${{ steps.chatops.outputs.SHA }}
FAILURE_FLAG: ${{ steps.ge.outputs.checkpoint_failure_flag }}
CHECKPOINTS
:
A comma separated list of checkpoint names to execute. Example: "checkpoint1,checkpoint2"NETLIFY_AUTH_TOKEN
:
A personal access token for Netlify.
NETLIFY_SITE_ID
:
A Netlify site id.
DEBUG
Setting this input to any value will allow the Action to exit with a status code of 0 even if a Checkpoint fails. This is used by maintainers of this Action for testing and debugging.
ACTION_DOCS_LOCATION
:
The absolute path where generated data docs generated by the Action run are located. This is useful if you want to deploy the data docs to an external service.
FAILING_CHECKPOINTS
:
A comma delimited list of failed checkpoints.
PASSING_CHECKPOINTS
:
A comma delimited list of passing checkpoints.
NETLIFY_DOCS_URL
:
The URL to the generated data docs on Netlify. This output is only emitted only if the input parameters NETLIFY_AUTH_TOKEN
and NETLIFY_SITE_ID
are provided.
CHECKPOINT_FAILURE_FLAG
:
This will return 0 if there are no Checkpoint failures and 1 if there are one or more Checkpoint failures.
This section is for those who wish to develop this GitHub Action. Users of this Action can ignore this section.
pip install -r requirements.txt
great_expectations init
to set up missing directoriesRun these commands from the repo root.
To see a Checkpoint pass, run great_expectations checkpoint run passing_checkpoint
To see a Checkpoint fail, run great_expectations checkpoint run failing_checkpoint
ValidationStore
is in use we may need to disable it so that the built Data Docs focus only on what is being validated in the Action without other side effects.Please feel free to jump into the Great Expectations Slack to ask us questions about the GitHub Action http://greatexpectations.io/slack