search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.09k stars 5.47k forks source link

Conformance tests #52775

Open max-arnold opened 5 years ago

max-arnold commented 5 years ago

I posted this on Slack, and @s0undt3ch suggested that it would be better to file an issue so it won't get lost.

The idea is to focus on adding low to medium effort / high impact tests, that would help to improve feature quality/parity. There are many things in Salt that could be invoked in multiple ways (or are pluggable) and are expected to work identically. Just a few examples:

  1. Feature parity between different commands like salt, salt-call, salt-ssh, salt-run salt.*, salt-run ssh.cmd. For example, missing wrappers for salt-ssh: https://github.com/saltstack/salt/issues/50196. What if some automated tool was able to run each module using all these salt-* commands and reported any inconsistencies/crashes?
  2. Ensuring that all cli utilies handle identical arguments in the same way, to prevent things like this: https://github.com/saltstack/salt/pull/50974, https://github.com/saltstack/salt/pull/53679
  3. Batch and non-batch APIs: https://github.com/saltstack/salt/issues/52762
  4. Ensuring that all modules conform to the same interface: https://github.com/saltstack/salt/issues/51809
  5. A test to detect accidentally introduced new hard dependencies (https://github.com/saltstack/salt/issues/53570#issuecomment-508691287)
  6. A pylint plugin to warn on direct imports: https://github.com/saltstack/salt/pull/50633#issuecomment-443579448
  7. Another pylint plugin to detect states that do not respect test=True (this is one of the basic expectations about Salt!). Just a rough example: comm -23 <(find salt/states -name '*.py' | sort -u) <(find salt/states -name '*.py' -exec grep -l -E '__opts__.*test' \{\} \; | sort -u )

These meta-tests (or conformance tests) could automatically check all new code to conform to existing coding conventions/invariants/architectural decisions. Two examples: https://github.com/saltstack/salt/pull/52368 https://github.com/saltstack/salt/pull/51900

I'm sure there is a lot of institutional knowledge that could be transformed into these kinds of tests that will assist in PR reviews.

max-arnold commented 5 years ago

Many of the things I mentioned also could be symptoms of code duplication.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.