samrocketman commented 3 months ago

Bug

When I run goss validate I expect a command to run; it does and returns the wrong output and bad exit code. This was expected because two tests fail. Fixing it is difficult because you can't see the error output of the actual command. All you see is the output was not expected but you don't get to see the contents of the output.

Reproduce

Here's a simple test.

goss add command "/bin/bash -exc 'mkdir /tmp/dir; touch /tmp/dir/afile'"

The purpose of this test is to give you an easily reproducible and simple error.

If you run goss validate it fails (expected). If you run rm -r /tmp/dir and then run goss validate it passes (expected). Run goss validate again and it fails (expected).

Missing from output

The problem is goss validate says something failed but not why. The following error should have shown up with stderr.

mkdir: cannot create directory ‘/tmp/dir’: File exists

Example bug output

However, goss validate only tells you the non-zero exit code and that some expected output was missing.

$ goss validate
F.F

Failures/Skipped:

Command: /bin/bash -exc 'mkdir /tmp/dir; touch /tmp/dir/afile': exit-status:
Expected
    1
to be numerically eq
    0
Command: /bin/bash -exc 'mkdir /tmp/dir; touch /tmp/dir/afile': stderr:
Expected
    "object: *bytes.Reader"
to have patterns
    ["+ mkdir /tmp/dir","+ touch /tmp/dir/afile"]
the missing elements were
    ["+ touch /tmp/dir/afile"]

Total Duration: 0.003s
Count: 3, Failed: 2, Skipped: 0

Related issues

814 marks this as solved but it might be incomplete.

I'm not sure if there's some option I'm missing on the command line but #814 does not work for #483 issue.

Environment

goss: goss version v0.4.4
os: Ubuntu 22.04.4 LTS

aelsabbahy commented 3 months ago

Hello, when goss is configured to match against an array, which is the default for backwards compatibility, it reads the output line by line but doesn't retain it.

To get the command output, you'll need to either:

A) modify your goss.yaml to be a string comparison:

command:                                                                                                                   
    /bin/bash -exc 'mkdir /tmp/dir; touch /tmp/dir/afile':                                                                 
        exit-status: 0                                                                                                     
        stdout: ""                                                                                                         
        stderr: |                                                                                                          
          + mkdir /tmp/dir                                                                                                 
          + touch /tmp/dir/afile                                                                                           
        timeout: 10000

B) Use array but run it in debug mode. This is useful if you don't need it in test results and it just prints to stderr.

goss -l debug v

Let me know if those two solutions work for you.