search

GothenburgBitFactory / bugwarrior

Pull github, bitbucket, and trac issues into taskwarrior
http://pypi.python.org/pypi/bugwarrior
GNU General Public License v3.0
734 stars 209 forks source link

Update Testing Infrastructure #833

Closed ryneeverett closed 2 years ago

ryneeverett commented 3 years ago

I saw the tox.ini in the repo and ran tox to run the tests. First, it only uses old versions of Py3 which I don't have, so I hacked in a py39-jira200. Having added that, loading the tests fails with #829, meaning no tests run.

Looking at the github actions, it seems like tox isn't used in CI, and CI doesn't even run python setup.py test like tox does. So maybe a place to start is bringing those two things in line?

Also, I noticed that CI is only testing on aarch64 and ppc4le. For a mostly-Python tool, that seems a bit esoteric, and especially so since it does not include the common x86 architecture. Maybe CI would run more quickly if the full matrix ran on x86 and only the latest python version / jira version ran on the other architectures?

Originally posted by @djmitche in https://github.com/ralphbean/bugwarrior/issues/831#issuecomment-894682505

ryneeverett commented 3 years ago

I saw the tox.ini in the repo and ran tox to run the tests. First, it only uses old versions of Py3 which I don't have, so I hacked in a py39-jira200. Having added that, loading the tests fails with #829, meaning no tests run.

Looking at the github actions, it seems like tox isn't used in CI, and CI doesn't even run python setup.py test like tox does. So maybe a place to start is bringing those two things in line?

I don't mind including support for tox if people find it helpful but I don't see much value in it and if it doesn't work I'd be inclined to remove it. I don't see any reason to test the full matrix locally. I'd rather let CI notify me of the occasions that a version-specific test failure occurs. I also don't believe it's advantageous to run tox in CI because we'd lose parallelism and the ability to easily view the matrix of checkmarks in CI.

I'd also note that I intentionally moved away from python setup.py test in part because it's deprecated.

Also, I noticed that CI is only testing on aarch64 and ppc4le. For a mostly-Python tool, that seems a bit esoteric, and especially so since it does not include the common x86 architecture. Maybe CI would run more quickly if the full matrix ran on x86 and only the latest python version / jira version ran on the other architectures?

Whoops. We should definitely be testing x86 too. I'm not necessarily opposed to a smaller matrix but I'm not sure CI time is really a problem. For the purposes of push-and-pray debugging I'm not sure it makes a difference because you're going to get feedback that CI failed as soon as one job fails.

djmitche commented 3 years ago

I suspect it takes longer to start jobs on platforms for which there is less availability? At any rate, if the project is still well within its actions budget then that's good news.

ryneeverett commented 3 years ago

I assume that github automatically starts jobs on platforms with higher availability first but I haven't confirmed it.

My interpretation of the new bait-and-switch actions budget is that only Ralph can see it since we're under his personal namespace, so I have no idea. If it becomes an issue I'd be inclined to consider switching if there's a CI service with a better free tier.