Automate Finding Test Crumbs

[john-science]

The problem we have had with recent PRs is that they create little "test crumbs". That is, new or altered tests create files that stick around in the repo area, but just show up in git as "untracked files".

This is messy and annoying for developers. But it is also a real hassle for various automated downstream processes.

We should automate a solution to this on GitHub CI.

And example solution would be:

• Run the ARMI unit tests on GitHub inside the git repo.
• Do something like: git clean -xnd to get a list of untracked files.
• If there are any "untracked files" in the workspace, fail the build.

This could be added directly into the tox.ini or in unittests.yaml.

[john-science]

@opotowsky @albeanth I am leaving this ticket here. This problem just bit us again, and I've always found it annoying. But, in retrospect, I bet this problem has an automatable solution.

[opotowsky]

Finding the files is a little more than git status, because often these files created during testing are already git ignored.

My testing protocol was try a thing, then run git clean -xf and if anything showed up as deleted, try something else and repeat. So if we run git clean -xf and ensure the stdout is empty, then that might work.

[opotowsky]

This isn't perfect. There was one or two directories I found (not in ARMI I don't think, but in another repo) in this process. But you don't want to run git clean -xfd becuse there will always be pytest/py_cache directories created, so that will always show up in the stdout.

[john-science]

Perhaps the solution is to have GitHub CI run git clean -xfd, and somehow parse the list of files/directories cleaned, and ignore any called "pytest".

I still think it's worth investigating. I spend a lot of time mentally trying to track when recent PRs create test crumbs. And Arrielle and I have both opened PRs to fix tests so they leave no crumbs.

[john-science]

Testing the current state of the repo:

I just checked out a fresh copy of the repo, ran all the unit tests, then did git clean -xnd in my work area. I found 67 untracked objects.

But I think these are all things we can ignore:

• 64 of the objects were /__pycache__/ directories, we can ignore those.
• .pytest_cache/
• armi/tests/tutorials/case-suite/ - questionable
• armi/tests/tutorials/logs/ - questionable

Of course, if we ran these on GitHub, we would also have /.tox/ directory that would be safe to ignore.

[opotowsky]

Just FYI the mpi tests leave behind their own logs files :

1. armi/logs/ARMI.armiRun.0001.stdout
2. armi/logs/armiRun.mpi.log