Define test matrix coverage

conda-incubator / conda-project

Tool for encapsulating, running, and reproducing projects with Conda environments

https://conda-incubator.github.io/conda-project/

BSD 3-Clause "New" or "Revised" License

28 stars 11 forks source link

Define test matrix coverage #118

Open mattkram opened 1 year ago

mattkram commented 1 year ago

We currently test across a range of conda and python versions. This means every time CI runs we run something like 80 parallel cases. We are also testing very old versions of conda.

I recommend we clearly defined what our supported versions are, and update our test matrix accordingly.

I recommend we test against:

Supported versions of conda
Supported versions of Python

And that we update this as those versions reach end of life and new versions become available.

mattkram commented 1 year ago

I think I'm searching for a different term here. I'm thinking of "what is our promised level of support"?

mattkram commented 1 year ago

Just looked and we are testing

9 versions of conda
3 operating systems
3 versions of Python

os: ["ubuntu-latest", "macos-latest", "windows-latest"] 
python-version: [3.8, 3.9, "3.10"] 
conda-version: ["4.9", "4.10", "4.11", "4.12", "4.13", "4.14", "22.9", "22.11", "23.1"]

I recommend we keep the Python and OS as is. For conda I recommend we remove old versions and support the last 1-3 versions, up for discussion. I believe conda now releases quarterly.

mattkram commented 1 year ago

Another option is to only test the oldest and newest versions of conda and Python supported before PRs are merged, and test the rest after merge into main. Theoretically we could have some errors and have to revert or change some merged behavior, but it may be rare enough to make it worthwhile?

AlbertDeFusco commented 1 year ago

That sounds reasonable. I've not had to implement any version specific code so far anyway.