search

scop / bash-completion

Programmable completion functions for bash
GNU General Public License v2.0
2.92k stars 380 forks source link

Rework test suite to additionally/primarily use captured outputs #573

Open scop opened 3 years ago

scop commented 3 years ago

Describe the feature/solution

Currently we pull a bunch of large containers for running the test suite, and run them against real commands in them. While this is useful in the sense that it tests against the very real things, and also in situations where unexpected errors in commands might occur, and tracks changes in developing distros, it is:

It would be great if we had a corpus of command outputs for different command versions on different platforms we would primarily test against. Secondarily we should (perhaps on schedule, bi-weekly or so) run the live tests (i.e. what we have now) at least against developing distros such as Fedora and Alpine bleeding edges, perhaps Debian too.

Maintenance (please complete the following information)

N/A

akinomyoga commented 3 years ago

I'm interested in how you think this should be achieved. For example, are target commands supposed to be replaced by the function of the same name in the tests? How are we going to manage the "corpus" of the command outputs? For example, if we save them in the git tree, I'm afraid that the repository size would become unreasonably large, so should we manage the "corpus" externally? Maybe we can automatically generate/update the corpus with CI.

scop commented 3 years ago

I was primarily thinking of adding mock executables, but guess functions could work as well. We have a few examples of the former in the tree, ls test/**/bin

Command outputs could be saved as text files, and the mocked executables/functions would basically cat them.

Sure, the tree would become largish, but I'm not sure how much of a problem that would be. But if it is, and an external storage would solve it, I'm all for it.

Generating in CI or some automated way would be nice. Probably need a way to make manual additions work too, though.