Allow test selection using a shell script

psss commented 3 years ago

@bachradsusi shared an interesting use case: In the selinux tests repository they are using a shell script to pick only those tests which have been modified by the pull request:

- name: Generate list of tests in this PR
  shell: |
      git log --format= --stat --name-only origin/master..HEAD | sed '/\//!d;s#\(.*\)/.*#\1#' | sort -u | xargs
  delegate_to: localhost
  register: tests_list

It would be nice to provide a similar functionality in the discover step. Currently we support selecting tests by name or filter:

discover:
    how: fmf
    test: /tests/basic
    filter: 'tier: 1'

We could also support executing a shell script instead:

discover:
    how: fmf
    test: shell(git log...)
    filter: 'tier: 1'

This brings, however, a similar question about security as mentioned in #373.

pvalena commented 3 years ago

Is this a generic case, or a special one?

psss commented 3 years ago

The specific use case related to choosing modified tests is now covered by #457.

teemtee / tmt

Allow test selection using a shell script #411