teemtee / tmt

Test Management Tool
MIT License
81 stars 122 forks source link

tmt t lint should try to cover adjust rules as well #898

Open lukaszachy opened 3 years ago

lukaszachy commented 3 years ago

I made typo in adjust rules, tmt tests lint didn't show any error: (enable vs enabled) Since adjust keys are defined (when, because and every tmt attribute for its object) we should be able to lint for unknown/typos

test: foo
adjust:
- when: distro == fedora
  enable: false
Athwale commented 3 years ago

Also when you forget the "when" keyword, lint crashes and you get a long traceback. The last line is fmf.utils.FormatError: No condition defined in adjust rule. That already looks like something that could be nicely printed as an error.

idorax commented 2 years ago

For adjust part,

test: foo
adjust:
- when: distro == fedora
  enable: false

self.node.get().keys() in tmt/base.py#L292 does return the L1 keys and doesn't return the L2 keys(e.g. when and enable), that's why tmt t lint failed to detect key enable is invalid (p.s. it should be enabled).

(Pdb) b tmt/base.py:291
Breakpoint 1 at /home/huanli/.virtualenvs/tmt/lib/python3.10/site-packages/tmt/base.py:291
(Pdb) c
/a
pass test script must be defined
pass directory path must be absolute
pass directory path must exist
warn summary is very useful for quick inspection
> /home/huanli/.virtualenvs/tmt/lib/python3.10/site-packages/tmt/base.py(291)lint_keys()
-> known_keys = additional_keys + self._keys
(Pdb) ll
289         def lint_keys(self, additional_keys):
290             """ Return list of invalid keys used, empty when all good """
291 B->         known_keys = additional_keys + self._keys
292             return [key for key in self.node.get().keys() if key not in known_keys]
(Pdb) pp additional_keys, self._keys
(['extra-nitrate',
  'extra-hardware',
  'extra-pepa',
  'extra-summary',
  'extra-task',
  'relevancy',
  'coverage',
  'adjust'],
 ['summary',
  'description',
  'contact',
  'component',
  'test',
  'path',
  'framework',
  'manual',
  'require',
  'recommend',
  'environment',
  'duration',
  'enabled',
  'order',
  'result',
  'tag',
  'tier',
  'link'])
(Pdb) pp self.node.get()
{'adjust': [{'enable': False, 'when': 'distro == fedora'}], 'test': 'foo'}
(Pdb) pp self.node.get().keys()
dict_keys(['adjust', 'test'])
# ^^^^^^^^^^^^^^^ It doesn't include key 'enable' and 'when' !!!

Hence, it looks we should get the L2 keys at tmt/base.py#L292.

idorax commented 2 years ago

To get L2 keys in property adjust, we can iterate self.node.get('adjust'), e.g.

(Pdb) p self.node.get()
{'adjust': [{'when': 'distro == fedora', 'enable': False}], 'test': 'foo'}
(Pdb) prop_adjust = self.node.get('adjust')
(Pdb) adjust_keys = [key for item in prop_adjust for key in item.keys()]
(Pdb) pp adjust_keys
['when', 'enable']
idorax commented 7 months ago

Here is a brief summary after discussing with TMT developers.