Improve CI Infrastructure support for Cantera

bryanwweber commented 4 years ago

Idea

Cantera uses GitHub Actions as our primary CI provider. We have identified several additional capabilities that would improve our productivity. This project will investigate, suggest, and implement improvements such as those listed above in our CI infrastructure.

Progress

Tests to run as part of CI within each PR

[x] mingw32 compiler on Windows
[x] Build and run the samples in the source repository as a regression test
[x] Run builds of our packaging infrastructure for conda, macOS, and Windows installer packages
[x] Test the MATLAB toolbox

Tests to run on the main branch after merging PRs

[ ] Test most recent versions of all dependencies
- [x] Python
- [x] Cython
- [ ] Sundials
- [ ] Eigen
- [ ] SCons
- [ ] fmt
- [ ] yaml-cpp
[ ] Real GCC compiler on macOS (can be installed by Homebrew, binaries are named gcc-9 and g++-9)
~~[ ] Run the Jupyter Notebook samples, maintained in a separate repository, as a regression test~~
[ ] Test the install scripts on platforms that our core contributors don't use as frequently (Linux distributions other than Ubuntu, primarily)
- [x] Tests on Fedora using a Docker container ~(not sure if this is possible on GH Actions)~ it is possible
- [x] Tests on Ubuntu pre-release version / non-LTS versions
- [ ] Tests on Debian using a Docker container
- [ ] Tests on archlinux using a Docker container
- [ ] Potentially other Linux flavors
[ ] Other tests of the installers

Difficulty

Easy

Required Knowledge

Experience with CI services preferred, although not required

Mentors

@bryanwweber

References

Cantera/cantera#775, Cantera/conda-recipes#22

bryanwweber commented 4 years ago

Copying To-Dos from https://github.com/Cantera/cantera/pull/775#issuecomment-585411101

Edit: moved to main post above

bryanwweber commented 4 years ago

It should be possible to test pre-release versions of Python on Ubuntu using the deadsnakes action: https://github.com/deadsnakes/action

ischoegl commented 4 years ago

@bryanwweber ... I'm not sure how converged the samples testing in Cantera/cantera#775 is (specifically for Python examples), but it definitely sounds like a good idea to run samples as part of CI tests.

See Cantera/cantera#821, Cantera/cantera#872, Cantera/cantera#873, Cantera/cantera#875, and Cantera/cantera#877. Thus far, I've only looked in the onedim and reactor folders, but there may be others.

ischoegl commented 4 years ago

I opened Cantera/cantera#892 to investigate running of samples during CI.

bryanwweber commented 3 years ago

Some further thoughts I have here, after about a year with GitHub Actions

[x] Run a separate workflow only when commits are made to main to do tests that take a long time but aren't critical prior to merging, for example, multiple versions of our dependencies, Jupyter samples, "nightly" builds of the Conda package, etc.
[ ] Run a separate workflow for benchmarking the code

ischoegl commented 2 years ago

Per Cantera/cantera#1287, the test removed in https://github.com/Cantera/cantera/pull/1287/commits/d0a343eca3df065d33821b961ee0cca39a7a1d1f should be reintroduced here.

speth commented 1 year ago

Cantera/cantera#1456 Introduced a workflow for "post-merge" tests, which currently consists of testing the latest Cython pre-release. We can add more jobs to this workflow that will help complete this enhancement. @bryan, would you mind if we moved the checklist from this comment into the initial post and used that to keep track of all the desired jobs?

One small issue with this workflow at the moment is the visibility of the results (or the lack thereof). The job status does get posted at https://github.com/Cantera/cantera/actions, and they do show up under the green check / red X for the latest commit at https://github.com/Cantera/cantera, but I'm not sure whether it will generate notifications when these tests fail -- some of the other workflows like the one for https://github.com/Cantera/pypi-packages that have been failing for a while don't generate notifications for me, even though I'm "watching" that repo.

bryanwweber commented 1 year ago

@bryanwweber, would you mind if we moved the checklist from this comment into the initial post and used that to keep track of all the desired jobs?

Sure, go for it!

even though I'm "watching" that repo.

I think you have to specifically opt-in to notifications about Actions results. https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/notifications-for-workflow-runs (although unhelpfully, that doesn't say where to opt-in).

Edit: Looks like you can only do it globally: https://github.com/settings/notifications

rwest commented 2 months ago

At the NumFOCUS summit came across this https://scientific-python.org/specs/spec-0004/ SPEC 4 — Using and Creating Nightly Wheels

Sounds reasonable to me - a way to check with "nightly" (their generic term for weekly, or whatever) versions of all (collaborating) dependencies, and let others check against (or just use) nightly Cantera builds.

bryanwweber commented 2 months ago

We use nightly wheels for our dependencies for post-merge tests: https://github.com/Cantera/cantera/blob/main/.github%2Fworkflows%2Fpost-merge-tests.yml#L20-L22

If someone wants to set up the pypi repo to push to anaconda, and get permission to use the scientific wheels upload spot, I'd be happy to help support that

Cantera / enhancements