As described in #16, the current GitHub Actions workflow to test the install of spokewrench is incomplete. The workflow (in .github/workflows/main.yml) runs a conda install based on the environment.yml file, which is locked to a single Python version. @LeeBergstrand posted the following comments about the current testing setup in https://github.com/rotary-genomics/spokewrench/issues/16#issuecomment-2116739686 :
We usually don't use environment.yaml to install a single Python application in production. Environment files are used as a stop-gap measure or for building a custom environment consisting of multiple pieces of software. Once the package is fully developed, Conda will use the data in meta.yaml instead to build and install the package. PIP uses pyproject.toml, and Conda's equivalent is meta.yaml. Ultimately, we could delete environment.yaml once we get the spokewrench package on Anaconda.org.
Usually, I will use more specific package versions in environment.yaml than meta.yaml. The meta.yaml tells Conda what versions are necessary to install the package alongside other Python software inside a single environment.
Generally, I don't lock to a specific Python version unless I am in the process of building a super complex pipeline like QIIME2 or Rotary. I generally provide a range of Python versions that work with the code with a low probability of errors. This increases compatibility with other software installed in the same Conda environment. For example, what if we want spokewrench to be installed alongside other software in the same Conda environment, but that piece of software requires Python 3.10? The installation would break... Providing a range of compatible Python versions solves this problem, as there will likely be an overlap between the package versions of each software.
Generally, I follow this design pattern.
I lock all the package versions during initial development to versions that are guaranteed to work.
I write unit tests and end-to-end tests.
I unlock the package versions and let the CI test the software with multiple Python versions.
We will need to overhaul the approach to testing in spokewrench, for example following the design pattern in 4 above, to test across a variety of Python (and other tool) versions.
As described in #16, the current GitHub Actions workflow to test the install of
spokewrenchis incomplete. The workflow (in.github/workflows/main.yml) runs a conda install based on theenvironment.ymlfile, which is locked to a single Python version. @LeeBergstrand posted the following comments about the current testing setup in https://github.com/rotary-genomics/spokewrench/issues/16#issuecomment-2116739686 :We usually don't use
environment.yamlto install a single Python application in production. Environment files are used as a stop-gap measure or for building a custom environment consisting of multiple pieces of software. Once the package is fully developed, Conda will use the data inmeta.yamlinstead to build and install the package. PIP usespyproject.toml, and Conda's equivalent ismeta.yaml. Ultimately, we could deleteenvironment.yamlonce we get the spokewrench package on Anaconda.org.Usually, I will use more specific package versions in
environment.yamlthanmeta.yaml. Themeta.yamltells Conda what versions are necessary to install the package alongside other Python software inside a single environment.Generally, I don't lock to a specific Python version unless I am in the process of building a super complex pipeline like QIIME2 or Rotary. I generally provide a range of Python versions that work with the code with a low probability of errors. This increases compatibility with other software installed in the same Conda environment. For example, what if we want spokewrench to be installed alongside other software in the same Conda environment, but that piece of software requires Python 3.10? The installation would break... Providing a range of compatible Python versions solves this problem, as there will likely be an overlap between the package versions of each software.
Generally, I follow this design pattern.
(Original post by @LeeBergstrand at https://github.com/rotary-genomics/spokewrench/issues/16#issuecomment-2116739686)
We will need to overhaul the approach to testing in
spokewrench, for example following the design pattern in 4 above, to test across a variety of Python (and other tool) versions.