Open pahaeanx opened 1 year ago
This is similar to (but better than!) an idea I had in the past, which was to add a 'description' attribute to tests that could be printed on test failure to give a bit more information to the user on which test failed. Being able to define it at test or file level, defaulting to the file name, would be handy when your tests are split over multiple gossfiles, possibly managed by multiple other sources.
correct me if I'm wrong but do title and meta cover the "hint"/"description" use case?
example file defining title and meta:
$ cat goss.yaml
command:
some_test:
title: This is some arbitrary title
meta:
any-key: any-value
version: 17
severity: very severe
contact: call bob
hint: node-exporter listening port
exec: |
echo "hi"
stdout:
- bye
example output:
$ goss v
FF
Failures/Skipped:
Title: This is some arbitrary title
Meta:
any-key: any-value
contact: call bob
hint: node-exporter listening port
severity: very severe
version: 17
some_test: exit-status: Error: Missing Required Attribute
Command: some_test: stdout: patterns not found: [bye]
Total Duration: 0.004s
Count: 2, Failed: 2, Skipped: 0
Just realized they're documented in the readme, but not the manual https://github.com/goss-org/goss#manually-editing-goss-files
Basically, title can be set to any string and meta can be set to any key/value (both are stored strings).
Oh wow, I somehow completely missed that.
After playing around with these a bit, the only criticism I have is that it seems that neither can be set at the file level?
# goss.yaml
title: testing file level titles
command:
fail_w/o_title:
exec: /bin/false
exit-status: 0
fail_w_title:
title: This test has a title set
exec: /bin/false
exit-status: 0
Output:
$ goss v
FF
Failures/Skipped:
Command: fail_w/o_title: exit-status:
Expected
<int>: 1
to equal
<int>: 0
Title: This test has a title set
Command: fail_w_title: exit-status:
Expected
<int>: 1
to equal
<int>: 0
Total Duration: 0.002s
Count: 2, Failed: 2, Skipped: 0
I guess it'd be nice to be able to set title
and meta
attributes on a file level that act as default values for tests that don't have these set.
The current limitation can be worked around by setting the same title
for each test (I'd simply set the file name) but that seems like a lot of unnecessary work and code.
Apart from that, that's exactly what I was looking for. Thanks!
@sshipway just want to confirm, does the current functionality of title/meta meet your suggestion above?
@pahaeanx I like your suggestion, but it might be a little (long) while before I get to it to be honest.
Mostly because, I'm thinking as a future enhancement Goss should have the ability to group tests into a test suite.. that suite can then have title/meta.. but it can also have other things like setup/teardown.
Perhaps the file will be the group.. or maybe there would be some other top level suite concept introduced that allows multiple suites to be set for a given file.
Once I get through the final Goss v4 release, I'll open up some issues mapping out the future roadmap. Hopefully, I can stimulate some OSS contributions to the projects for some of them.
I'd be great if there were a way to either add information to tests (such as a name hint) or showing their origin file in case they are pulled in via
gossfile
.For example, I split up most tests into what I call group files and test files. Group files simply pull in all tests for a particular group (ansible in this case). Like so:
Where each of these files contain tests for the component they are named after (
verify_node_exporter.yaml
for the Prometheus node-exporter, etc). This allows test files to be nice and contained but still allows us to quickly and easily add a new groups or tests.These test files then get templated to a host and validation is done using
Anyway, the problem arises when one or more tests fail. A user has no easy way of telling where the failing test originates from.
Ultimately this is a problem of information: I'd like some way to know more about the failed test. Not everyone knows the system by heart. Say the port/listening test for port 9100 failed -- there's no immediate way of knowing what is supposed to listen on there.
It is fair to question whether this should be part of a test suite, but I think that even a little extra information would be enough to allow immediate targeted troubleshooting.
Describe the feature:
I'd be great if we can see the file a failed test originates from. Something like this (extends the
documentation
format).or even some kind of hint we can set, possibly on a per-test basis:
Both of these solutions immediately give enough additional information to for targeted troubleshooting.
Describe the solution you'd like
This boils down to what is feasible. I guess the most simple solution would just to display the test origin in the final result. This would already be a huge improvement and sufficient for most situations.
Ideally there would be some kind of optional
hint
resource which can be used to add additional information to all tests contained in a file. This could also be set on a per-test basis:If these fail, one would display
Hint: node-exporter tests
(the default set viahint
resource), the other would displayHint: node-exporter listening port
.Describe alternatives you've considered I don't think there is any other solution or workaround for this problem at the moment.