Adjusting distro to disable the test did not work

guoguojenna commented 9 months ago

My intention is to disable this test when distro < rhel-10, so define the fmf file as below. When I run the test on RHEL-9.4.0-20231109.18, expect the test can be disabled and not run any more. The actual result is it still can be run normally.

$ cat main.fmf 
summary: KS-RDMA-TC:pyverbs-tests-2
description: pyverbs-tests-2
contact: Zhaojuan Guo <zguo@redhat.com>
component:
  - rdma
framework: shell
test: bash -x ./runtest.sh
require: 
  - python3-pyverbs
enabled: true
adjust:
    enabled: false
    when: distro < rhel-10
    because: The feature added in rhel-10
duration: 10m

Run command: $ tmt run discover -h fmf provision -h connect -g rdma-qe-41.rdma.lab.eng.rdu2.redhat.com -u root -p redhat execute -h tmt tests -n "rdma/pyverbs-tests-2" plans --default -d -vvv

guoguojenna commented 9 months ago

Then I change the enabled to false and adjust condition, and then run on RHEL-9.4.0-20231109.18,

Actual result: test was disabled and not run Expected: test was enabled and run

enabled: false
adjust:
    enabled: true
    when: distro >= rhel-8
    because: The feature was added in rhel-8

lukaszachy commented 9 months ago

Hi @guoguojenna, the problem is that user is expected to provide the "context". There is no auto detection (#771). So to make these rules work you need to use tmt -c distro=rhel-9.4.0 run .... on the commandline or have context provided in the API calls for testing farm.

CI as packit set the context (distro, arch, trigger) already.

guoguojenna commented 9 months ago

Then I change the enabled to false and adjust condition, and then run on RHEL-9.4.0-20231109.18,

Actual result: test was disabled and not run Expected: test was enabled and run

enabled: false
adjust:
    enabled: true
    when: distro >= rhel-8
    because: The feature was added in rhel-8

tmt -c distro=rhel-9.4.0

Good to know, thank you!

teemtee / tmt

Adjusting distro to disable the test did not work #2470