dbt-labs / dbt-snowflake

dbt-snowflake contains all of the code enabling dbt to work with Snowflake
https://getdbt.com
Apache License 2.0
296 stars 176 forks source link

[CI/CD Improvements] Update and simplify CI workflows #1163

Closed mikealfare closed 3 months ago

mikealfare commented 3 months ago

Is this your first time submitting a feature request?

Describe the feature

As a maintainer, I want to simplify the CI jobs, so that I know I am running the same CI regardless of scenario (e.g. PR vs scheduled tests vs end to end tests for dependent packages).

## Acceptance Criteria
- [ ] All workflows adhere to the style standards below
- [ ] One CI workflow runs all checks: integration tests, unit tests, code quality tests, build verification
- [ ] Branch protection rules are limited to a single job, regardless of OS, python version, etc.
- [ ] I can manually run a CI workflow; optionally, I can provide specific branches for dbt dependencies (`dbt-adapters`, `dbt-common`, etc.) similar to what is done with `integration-tests.yml` today; the current workflow is cancelled in favor of the new workflow if the same run is triggered with the same branches
- [ ] I can trigger a CI workflow from another repo for end to end testing, similar to what is done with `integration-tests.yml` today
- [ ] A CI workflow runs on every PR; the current workflow is cancelled in favor of the new workflow when the PR is updated
- [ ] A CI workflow runs against each protected branch (`main`, `1.8.latest`, etc.) every week; the current workflow is cancelled in favor of the new workflow if the CI workflow is triggered for the same branch (shouldn't happen since the workflow should finish in far less than a week)
- [ ] I'm able to easily see what triggered each CI workflow in the Actions tab (e.g. don't use a generic run name or job name of "Adapter Integration Tests")
- [ ] Integration tests are only configured in one place (e.g. environment variables, etc.)
- [ ] Legacy workflows are removed (e.g. `code-quality.yml`, `integration-tests.yml`, `unit-tests.yml`, etc.)

Suggestions

While these workflows all do roughly the same thing, they are different enough (e.g. concurrency, updating upstream branches, alerting, etc.) that it might be easier to manage as separate workflows.

Style standards:

mikealfare commented 3 months ago

This issue is consolidated into https://github.com/dbt-labs/dbt-snowflake/issues/1004.