search

goss-org / goss

Quick and Easy server testing/validation
https://goss.rocks
Apache License 2.0
5.58k stars 470 forks source link

V4 new #814

Closed aelsabbahy closed 1 year ago

aelsabbahy commented 1 year ago
Checklist

Description of change

Overview

This PR is a combination of v4 changes + a bunch of misc fixes/additions. My hope is to get this released sooner rather than later so future PRs/changes can be smaller and incremental.

The major changes with v4 is the ability to compare different types and more advanced string matching

note: Manual has been updated to document most (if not all) new matchers, also testdata/ folder has ~80% coverage of the matcher language.

For example:

command:
  echo_test:
    # command that outputs JSON, but this could be anything, http response, command output, etc
    exec: |
      echo '{"string_value": "15"}'
    exit-status: 0
    stdout:
      # advanced string parsing
      gjson:
        # extract "string_value"
        string_value:
          and:
            # The value is numerically <= 20 (auto type conversion)
            - le: 20
            # The value is numerically 15
            - 15
            # Equal does a strict check, the types have to match, hence string_value is "15" but not 15
            - equal: "15"
            - not: {equal: 15}

This conversion also allows treating an io.Reader (memory efficient line-by-line parsing) as a whole string (when compared with a string). This has the benefit of showing the full command output, which was a long-requested feature. For example:

$ cat goss.yaml
command:
  echo_test:
    exec: |
      echo 'hello world'
    exit-status: 0
    stdout: |
      goodbye world

$ goss v
.F

Failures/Skipped:

Command: echo_test: stdout:
Expected
    "hello world\n"
to equal
    "goodbye world\n"

Total Duration: 0.001s
Count: 2, Failed: 1, Skipped: 0

Changes

Decisions

Should -o include_raw be enabled by default, and the option becomes -o exclude_raw

$  cat goss.yaml
command:
  echo_test:
    exec: |
      echo '{"string_value": "15"}'
    exit-status: 0
    stdout:
      gjson:
        string_value:
          le: 5

Without include_raw:

$ goss v
.F

Failures/Skipped:

Command: echo_test: stdout:
Expected
    15
to be numerically le
    5
the transform chain was
    [{"gjson":{"Path":"string_value"}},{"to-numeric":{}}]

Total Duration: 0.001s
Count: 2, Failed: 1, Skipped: 0

With -o include_raw:

$ goss v -o include_raw
.F

Failures/Skipped:

Command: echo_test: stdout:
Expected
    15
to be numerically le
    5
the transform chain was
    [{"gjson":{"Path":"string_value"}},{"to-numeric":{}}]
the raw value was
    "{\"string_value\": \"15\"}\n"

Total Duration: 0.001s
Count: 2, Failed: 1, Skipped: 0

reverted (defer to later)

These were changes living on the v4 branch. However, they are either bad fixes, or need more polish before release

petemounce commented 1 year ago

Should -o include_raw be enabled by default, and the option becomes -o exclude_raw

I would prefer -o include_raw to be the default, because I think that's a higher UX. I would be frustrated if I got a failure, had to re-run with -o include_raw and the 2nd run passed; indicating the test is flaky somehow. I think that it's best to give the data up-front, because in the context of machine-assertions (and especially when some can be remote-assertions like http, addr) the chance of flakes is higher than in "normal" test frameworks.

petemounce commented 1 year ago

(Also - this is great work; thank you!)

aelsabbahy commented 1 year ago

Awesome, thanks for the thorough feedback. I'll address it over the coming week and hopefully this gets merged/released after that.

I would prefer -o include_raw to be the default

This was my leaning as well, for all the reasons you mentioned.

aelsabbahy commented 1 year ago

Addressed some (most/all?) of the feedback here.

@petemounce let me know if all looks good, if I don't hear back in a week, I'll merge it.

aelsabbahy commented 1 year ago

Modified install.sh (locked down version) to avoid breaking anyone. Merging, will create a release candidate release.