Whenever any sample program is added or modified, all of the tests run. This is not only an inefficient use of resources and time, it is also confusing for the user who doesn't know our workflow and is left with a blinding wall of text that they do not know how to interpret. For example, see the last 2 or 3 comments in #3953 .
Possible Solution
Only run the necessary tests based on what is in the PR:
If any of these files change, run the entire test suite:
.glotter.yml
pyproject.toml
poetry.lock
.github/workflows/test-suite.yml
Whatever automation script is used to determine what needs to be run
Else if any testinfo.yml file changes, run all the tests for those languages
Else if any sample programs added/modified, only run the tests for those sample programs
Else, indicate nothing needs to be done and exit with a zero status
Problem
Whenever any sample program is added or modified, all of the tests run. This is not only an inefficient use of resources and time, it is also confusing for the user who doesn't know our workflow and is left with a blinding wall of text that they do not know how to interpret. For example, see the last 2 or 3 comments in #3953 .
Possible Solution
Only run the necessary tests based on what is in the PR:
.glotter.ymlpyproject.tomlpoetry.lock.github/workflows/test-suite.ymltestinfo.ymlfile changes, run all the tests for those languagesThis solution can run weekly in a cron schedule.