search

AntelopeIO / DUNES

Docker Utilities for Node Execution
Other
26 stars 20 forks source link

Periodically Run Expensive Tests #211

Closed ScottBailey closed 1 year ago

ScottBailey commented 1 year ago

202 / #208 test the image bootstrapping system. A subset of tests is all that should run on a per check in basis.

The suffix expensive was added to the test name (e.g. test_cdt2_expensive) as an indicator that maybe it shouldn't run on every commit. Additionally, the argument -k not expensive to the workflow pylint command.

@mikelik correctly makes the argument that disabling these tests in CI can allow a regression to slip by us that would otherwise be caught.

One option to resolve this is to run the tests periodically. Maybe once a week or even once a month.

ScottBailey commented 1 year ago

Nightly and on merge?

pytest -v --run-all

ScottBailey commented 1 year ago

After considering this a bit more, it seems like the following trigger - or something similar - is probably the correct option:

on:
  push:
    branches:
      - main
      - "release/*"
  workflow_dispatch:

This should allow for the full run when:

  1. a push to main occurs
  2. a push to a release branch occurs
  3. the user requests the long build
ScottBailey commented 1 year ago

Rather than run periodically, we should run the extended tests when a PR is merged into main or release*.