psathyrella
commented
2 years ago This all sounds great, thanks. I just gave you collaborator access. The current testing uses test/test.py, which runs several partis commands, writing results to test/new-results/, then compares them to test/ref-results/ and prints some ascii art output showing the differences (here I screwed with some numbers to make things different):
So this is comparing e.g. annotation + clustering performance, log probabilities, and run times. Test.py has --paired for paired h/l data, and --slow to run a slower (but more detailed/accurate) set of tests. So if I just want to see if anything got broken, I just without --slow, but if I've made some major changes and want to see how much they affect actual numbers I run with --slow. So full testing is running all 4: ./test/test.py --run-all. If I've made a bunch of changes that modify results, and I'm convinced I want to commit it, I run ./test/test.py --bust-cache (which replaces the reference results) and commit.
So if you wanted to automate this you'd probably want to have test.py fail if there's differences I'm guessing? Or something like that.
I'd also be really excited to completely switch to quay from dockerhub, for which the first step sounds like you've mostly got running -- have a build on quay -- but also requires editing the docs to point people at that quay, and also probably figuring out a way to auto update dockerhub since people will be pulling from there for a long time after we say to start using quay, so I've started a new issue
I've been building an image locally and pushing it to quay.io/matsengrp/partis - but I think there might be a few helpful github actions which could help with autobuilds and possibly CI for PR's and commits using your current unit testing infrastructure. I would be happy to throw this on a TODO list somewhere. Giving me access to push might help - otherwise I'm happy to submit PR's from a fork.