Open davidak opened 2 years ago
abathur
commented
2 years ago I am not really sure how dressing it up for ~humans is done, but maybe emitting TAP is the right first step? I feel like I remember reading that there are some tools you can just pipe TAP through to get something humane?
thufschmitt
commented
2 years ago I was about to suggest re-opening that on nixpkgs, but then I saw that it’s where you come from :D (thanks for the gratuitous shot btw).
One fundamental issue here is that Nix doesn’t know anything about “tests” − it only builds derivations, whatever they represent. And you probably wouldn’t expect the same output when building a package and running a test (and teaching core Nix about tests is probably not a good idea for a number of reasons).
But the new Nix cli does have a (weak) notion of test thanks to nix flake check and the checks flake outputs. Maybe that could be generalized to non-flakes with a more generic nix check command (still a bunch of things to define wrt what the semantics should be, but nothing too hard I think), and be made so that this specific command has a different output for its toplevel goals.
So nix build --file nixpkgs seaweedfs would build seaweedfs using the progress bar, and nix check --file nixpkgs seaweedfs would build seaweedfs.tests, and display the build result of these in a different manner, probably somewhat similar to what you’re suggesting (or export it as TAP, or whatever. But do something different).
One issue with the above though, is that “tests” can mean a lot of different things for a Nix derivation. If my derivation just runs a simple command and expects it to succeed (like in your example), the above makes sense. But if it actually runs a full testsuite using a test framework that already has its own way of printing success and failure (maybe outputting in TAP format itself), then we might be better served by just letting the tool output what it wants and leaving the Nix output out of the way as much as can be.
davidak
commented
2 years ago “tests” can mean a lot of different things for a Nix derivation
here i was just talking about stuff defined in passthru.tests which i call "package tests" (that's how i documented it), but there can also be "nixos tests"
checkPhase is just part of build and should be hidden from end-users unless an error happens
currently we probably only have 1 or 2 tests per package, so my suggested output UI might not make a lot of sense. but it should be less verbose
i'm overall frustrated with the input and output UI of Nix from an enduser perspective. which starts with it's being experimental for years, not easy to use and understand and much of that might be due to the fact that nix does not know about concepts used in nixpkgs like packages, tests, ... on a low level (i have to rething that. maybe i just start a "friendly-nix" python wrapper that allows nix install python and makes me happy and don't bother you anymore)
thufschmitt
commented
2 years ago it should be less verbose
Out of curiosity, would the current “progress-bar” of the new CLI work nicer you think? (IIRC you can try it with the old cli by passing --log-format bar -q)
davidak
commented
2 years ago That does not work:
[davidak@gaming:~/code/nixpkgs]$ nix-build nixpkgs/ -A seaweedfs.tests --log-format bar -q
error: unrecognised flag '-q'
Try 'nix-build --help' for more information.
[davidak@gaming:~/code/nixpkgs]$ nix-build nixpkgs/ --log-format bar -q -A seaweedfs.tests
error: unrecognised flag '-q'
Try 'nix-build --help' for more information.
[davidak@gaming:~/code/nixpkgs]$ nix-build --log-format bar -q nixpkgs/ -A seaweedfs.tests
error: unrecognised flag '-q'
Try 'nix-build --help' for more information.
[davidak@gaming:~/code/nixpkgs]$ nix-build --log-format bar nixpkgs/ -A seaweedfs.tests
error: getting status of '/home/davidak/code/nixpkgs/nixpkgs': No such file or directory
[davidak@gaming:~/code/nixpkgs]$ nix-build nixpkgs/ --log-format bar -A seaweedfs.tests
error: getting status of '/home/davidak/code/nixpkgs/nixpkgs': No such file or directoryHow would i run the tests in nixpkgs/ with the new CLI with progress-bar?
I can run this command, but get no output at all:
[davidak@gaming:~/code/nixpkgs]$ nix build -f . --log-format bar seaweedfs.tests
So i don't know if it has run any test and if it succeeded or if nix is just broken and does nothing. Because the part that would give a user a useful feedback is broken, they could assume nix is also broken.
thufschmitt
commented
2 years ago [davidak@gaming:~/code/nixpkgs]$ nix build -f . --log-format bar seaweedfs.tests
So i don't know if it has run any test and if it succeeded or if nix is just broken and does nothing. Because the part that would give a user a useful feedback is broken, they could assume nix is also broken.
Yup’ that’s a possibility (although you’re in a slightly edge-case where everything is either built locally or builds to fast for the progress-bar to have time to show-up) . As I said, it’s still experimental and probably not meant to stay as-it-is. I was more interested about what you’d like to see in the failing case
davidak
commented
2 years ago I was more interested about what you’d like to see in the failing case
In the best case every check would have a useful output that explains why it failed, for example a string not matching. Similar to unit tests.
Is your feature request related to a problem? Please describe.
Currently it looks like this:
There is a lot of clutter and the information that i want is hidden (which tests was run) or not there (where the tests successful?).
Describe the solution you'd like
It should look like this:
✘ should be red ✔ should be green there can be a yellow symbol for timeout or similar status where the test did not run for some reason
This is how a tool i made for my company looks like: https://raw.githubusercontent.com/davidak/test/master/demo.svg
Related:
Describe alternatives you've considered
Additional context
Of course every output of Nix should look user friendly like this, so it does not scare users away.