search

teemtee / tmt

Test Management Tool
MIT License
77 stars 117 forks source link

`tmt` doesn't fail on invalid YAML keys #3063

Open abitrolly opened 1 week ago

abitrolly commented 1 week ago

The following main.fmf definition is invalid, but it doesn't stop ./test.sh from executing and failing.

summary: Test `tmt --help` output
test: ./test.sh
requires: tmt

test.sh is standard from template.

#!/bin/sh -eux

tmp=$(mktemp)
tmt --help > "$tmp"
grep -C3 'Test Management Tool' "$tmp"
rm "$tmp"

And here is the output.

$ tmt run --all provision --how container
/var/tmp/tmt/run-028

/default/plan
    discover
        how: fmf
        directory: /data/s/gitlab-ai-gateway/xxx
                warn: /test01: - 'requires' does not match any of the regexes: '^extra-'
        summary: 1 test selected
    provision
        queued provision.provision task #1: default-0

        provision.provision task #1: default-0
        how: container
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 40 (Container Image)

        summary: 1 guest provisioned
    prepare
        queued push task #1: push to default-0

        push task #1: push to default-0

        queued prepare task #1: requires on default-0

        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        where: default-0
        package: /usr/bin/flock

        queued pull task #1: pull from default-0

        pull task #1: pull from default-0

        summary: 1 preparation applied
    execute
        queued execute task #1: default-0 on default-0

        execute task #1: default-0 on default-0
        how: tmt
        progress:              

        summary: 1 test executed
    report
        how: display
        summary: 1 error
    finish

        container: stopped
        container: removed
        container: network removed
        summary: 0 tasks completed

total: 1 error
LecrisUT commented 6 days ago

I guess you are referring to the json schema validation. It is tricky because tmt does not yet expose json schemas for plugins which would make even the example plugin un-runnable. #2946 can solve that and maybe after that the json schema validation can be promoted to required.