saucelabs / saucectl

A command line interface for the Sauce Labs platform.
Apache License 2.0
42 stars 16 forks source link

[enhancement] Make saucectl validate YML #74

Closed dpgraham closed 3 years ago

dpgraham commented 4 years ago

The problem

It would help usability of saucectl if we validated the contents of the config.yml and gave helpful error messages

Example 1: Leaving the "files/" attribute blank in the saucectl config makes saucectl run not do anything

Steps to reproduce

git clone git@github.com:dpgraham/testbed.git && cd testbed
npm install
saucectl run --config .sauce/no-files-config.yml
# .sauce/no-files-config.yml
apiVersion: v1alpha
metadata:
  name: Feature XYZ
  tags:
    - e2e
    - release team
    - other tag
  build: Release $CI_COMMIT_SHORT_SHA
suites:
  - name: "saucy test"
    match: ".*.(spec|test).js$"
image:
  base: saucelabs/stt-cypress-mocha-node
  version: latest
sauce:
  region: us-west-1
  sauce-cypress-runner git:(devx-269-multi-reporters) saucectl run
➜  sauce-cypress-runner git:(devx-269-multi-reporters) 

^ Running saucectl on the above config shows no output

Recommended solution: Make a validator in Go that checks the YML object before processing it and then gives helpful error messages if it wasn't parsed properly.

Example 2: Providing a string when it's expecting a string array throws a GoLang error

If you provide a string when it should be an array it gives a GoLang error like this failed to parse job configuration: yaml: unmarshal errors:\n line 9: cannot unmarshal. This may be confusing to users, so I think we need to add a YML validation step that gives human readable errors when the user provides the wrong format so it says something like found a string in the "files:" attribute when we expect an array of strings.

Ideally we should accept strings and string[] and just treat strings as a singleton array.

Reproduce

git clone git@github.com:dpgraham/testbed.git && cd testbed
npm install
saucectl run --config .sauce/string-files-config.yml 
# Output
5:03PM ERR failed to execute run command error="failed to parse job configuration: yaml: unmarshal errors:\n  line 9: cannot unmarshal !!str `cypress/` into []string"

alexplischke commented 3 years ago

Addressed via the release of IDE Integrations (e.g. https://docs.saucelabs.com/testrunner-toolkit/ide-integrations/vscode).