CICD Config design

[tjduigna]

Context: See #136, #161, #209 and #214 for initial justifications for the following unified issue. Specifically from #161: Easily tracked code coverage is an absolute requirement for the new CICD flows

We have the following CICD-related configuration files:

• .travis.yml - does basically everything, build, test, publish conditionally on master
• appveyor.yml - builds + tests on windows machines
• meta.yml - conda packaging config (is this correct?)
• requirements.txt - specifies the python stack
• setup.py - importantly installs JS extensions

Github actions appear to have support for job matrices (multiple python versions), virtual environments (multiple OSes including windows), and can execute scripts from the git repository so that build logic can be put in a script (or scripts) and recycled in local development and in CICD. Instead of scripts, we could opt for a Makefile with commands to be invoked locally or in CICD. This removes the need of .travis.yml and appveyor.yml in favor of .github/workflows/branch.yml and .github/workflows/master.yml. To further reduce code duplication across workflows, we can leverage action composition as detailed here https://github.blog/changelog/2021-08-25-github-actions-reduce-duplication-with-action-composition/

The meta.yml file is used by conda packaging mechanics and I believe we want to continue supporting conda packaging as it is the easiest manager of the pydata stack. That leaves requirements.txt and setup.py. Though pip has made a lot of progress in the last few years, it is not really the most portable package management solution. I would recommend poetry but I don't think it plays that nicely with conda so it would be nice to hear other thoughts about the python packaging tools.

[tjduigna]

We can avoid dependency bloat by some appropriately categorized requirements.*.txt files. It looks like we can have the following:

• requirements.txt # user dependencies
• requirements.dev.txt # developer tools
• requirements.publish.txt # for publishing docs
• requirements.release.txt # for releasing conda packages

The dependencies are largely hard-coded in the existing CICD workflows so externalizing them to requirements files cleans things up a bit.

[tjduigna]

With #237 and #239 - #245 ++ the overall port to github actions can be considered more or less complete. Thoughts so far:

1. the cicd-matrix.py script is a bit janky. I thought we could take a composite action and make a single step job that just publishes the os and python-version arrays, moving this bit of logic into the workflows directory, but a quick test of this was unsuccessful.
2. There is a duplication in the requirements specification via requirements.txt for pip-based installs and the meta.yaml for the conda build process. We already update the version in this file in the pipeline, perhaps it makes sense to also cat the requirements file into the yaml config.