Consider alternative test runner

NixOS / nix

Nix, the purely functional package manager

https://nixos.org/

GNU Lesser General Public License v2.1

12.96k stars 1.54k forks source link

Consider alternative test runner #761

Open ctheune opened 8 years ago

ctheune commented 8 years ago

The current test runner is a bit, clunky, I'd love to have one that

spills fewer stuff on the console on succesfull runs
simplifies test registration / maintenance
allows selectively running and analyzing tests

I have started an implementation using pytest - I'll upload my branch a bit later.

TODO

[ ] get general buy-in
[ ] fix failing tests
[ ] documentation

ctheune commented 8 years ago

Alright, there's a PR for that.

To try it out, run "make installcheck" or, do things like "python -m pytest" to run them manually.

Interesting options are:

"python -m pytest --lf" -- runs only the tests that are still failing (or all if none failed previously)
"python -m pytest -k lang" -- runs only the tests with "lang" in their name (basically the filename)

fkz commented 8 years ago

generally :+1: I like it even more if I get a nice IDE-integration/tool where I get nice green and red boxes I can ckick on to jump to the right tests and a nice progress bar etc., is there something like that for pytest?

In light of #341 I'm not sure we want to add an other dependency though (python). It's probably okay for now, especially since it's only used for tests.

fkz commented 8 years ago

hopefully this will motivate me to finally provide test coverage for #709 :smiley:!

ctheune commented 8 years ago

Right - it's only a dependency at build time - not a runtime dependency. And the tests do not have to start being written in Python (although some refactoring might help isolation).

Pytest has a huge amount of plugins, I don't like the progress bar plugin much, but developers can customize this to their liking.

fkz commented 8 years ago

cc @edolstra @shlevy any thoughts on this one?

shlevy commented 8 years ago

Don't care either way personally.

edolstra commented 8 years ago

I'm not necessarily opposed, but I'm a bit reluctant to introduce a dependency on Python, even if it's only at build time, just to get prettier test output. Note that you can run individual tests currently, just do bash ./bla.sh in the tests directory.

Can pytest run tests in parallel? That would be a compelling advantage. (Would require some changes to our tests though.)

copumpkin commented 8 years ago

While we're increasing our c++ footprint, why not just pull in https://github.com/google/googletest?

edolstra commented 8 years ago

@copumpkin Good point...

ctheune commented 8 years ago

Yes, it can run things in parallel.

I absolutely understand any general reluctance. And I'm happy to go with any other runner -- pytest is my favorite and what I'm comfortable with.

There are more advantages than I can list right away, but I found the readability of the current setup atrocious. :)

Something I also worked on my branch is to allow way more fine-grained selection and reporting of errors in the eval- and parse- tests by making them first-level test types in pytest.

I won't be able to help with googletest, I think, but that shouldn't stop us - I'd just rather have a better test runner, not my favorite toy. If you like me to contribute then I'll be happy to learn, but also happy to bring what I know. :)

vcunat commented 8 years ago

Well, it's a build-time dependency iff you want to run tests, isn't it? I don't think that would limit portability. You just first build nix without tests and then you can use it to build tested nix from nixpkgs.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

stale[bot] commented 2 years ago

I closed this issue due to inactivity. → More info