CI/CD: Automated testing with pytest and mocks

[evanroyrees]

Continuous Integration / Continuous Deployment (CI/CD)

Provide unit tests and integration tests using pytest and a mock library (could be built-in unittest.mock, pytest.monkeypatch, pytest-mock, etc.)

After implementation of these tests:

1. 🛠️ 🧰 the pipeline can be continuously tested (automatically) with each pull request and merge. Multiple CI services such as travis-CI, circle-CI, etc. offer their service for free if you are an open source project.
2. ✅ ❗ ❌ These tests can then be reflected in each PR, similar to how our documentation is now checked with each PR.
3. 🐛 This will save us the effort of having to manually look through any changes in the code for possible introductions of bugs to the pipeline.
4. 🎨 Writing these tests requires a slight shift in mindset where exceptions need to be considered before-hand.

Resources:

• Raphael Pierzina pytest blog post (part 1) - link I would highly recommend starting here if you are unfamiliar with pytest. This is a wonderful post with many practice examples.
• medium post using pytest and pytest-mock - link
• good pytest practices - link
• realpython.com - mocking
• schedule lib tests
• intro to mocking in python
• (medium post) Showing some basic mocking usage
• Python Mocking 101: Fake It Before You Make It
• (medium post) A few simple examples explaining when and how to use mock
• mocking API calls (nice example of calling external tools)
• gist showcasing some python mocking
• Detailed examples of mocking in python
• blog post discussing mocking regular functions (and a mention of a common gotcha at the beginning)
• hackersoft.io post on mocking (includes section on mocking calls to third-party tool)
• 40 examples showcasing the mock.patch method
• (video) Testing the untestable: a beginner’s guide to mock objects
• Another guide on mocking in python
• pytest: How to mock in Python (blog post)

Legend

Work-in-progress (WIP) 🛠️

Data for coverage and quick CI/CD

Contigs containing annotations:

• [x] Eukarya, Viruses, Bacteria, Archaea, Unclassified - taxonomy-specific
• [x] Bacterial markers - marker-specific (43 contigs contain all unique 139 markers)
• [x] Archaeal markers - marker-specific (8 contigs contain all unique 164 markers)
• [x] Contigs with pre-assigned target genomes - binning assessment (determinism)

Associated data to mock for CI/CD

• [x] blast output - skip diamond blastp
• [x] blast hits assigned taxids - skip parsing accession2taxid.gz
• [x] Read alignments to contigs - skip read alignment
• [x] ncbi databases - skip download/formatting of databases
• [x] marker database - skip download/formatting of marker databases
• [x] hmmscan table - skip search with hmmscan

Unit test implementations

Common

• [x] test_metagenome.py 🛠️
• [x] ~test_metabin.py~
• [x] test_coverage.py
• [x] test_kmers.py
• [x] test_markers.py

External

• [ ] test_diamond.py
• [ ] test_prodigal.py
• [ ] test_bowtie.py
• [ ] test_samtools.py
• [ ] test_hmmer.py
• [ ] test_bedtools.py

Config

• [ ] test_user.py
• [ ] test_project.py
• [ ] test_databases.py
• [ ] test_environ.py

Taxonomy

• [ ] test_lca.py
• [ ] test_majority_vote.py
• [ ] test_ncbi.py
• [x] test_vote.py

Binning

• [x] test_recursive_dbscan.py
• [x] test_unclustered_recruitment.py
• [x] test_summary.py

Integration test implementations

• [ ] test_imports.py
• [ ] test_entrypoints.py
• [ ] test docker
• [ ] test bioconda build

[evanroyrees]

Note: For adding pre-commits with CI:

💚 https://pre-commit.com/#usage-in-continuous-integration