search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
345 stars 343 forks source link

Runnable: support own configuration and suite configuration #5998

Closed clebergnu closed 3 months ago

clebergnu commented 3 months ago

Describe the bug Currently, as a convenience, a Runnable instantiated without a config, is given a configuration that matches its kind. Example:

from avocado.core.nrunner.runnable import Runnable
>>> Runnable(kind='noop', uri=None).config
{'runner.identifier_format': '{uri}'}

Also currently, the suite will overwrite a runnable's config with its own.

The problem with this approach is that when a Runnable is used in a Suite with its own configuration, it's not possible to distinguish what was originally an empty configuration (that was turned into the default one).

Steps to reproduce Create a noop_config.json file such as:

{"kind": "noop", "uri": "noop", "config": {"runner.identifier_format": "nothing-op"}}

Run: avocado run noop_config.json

Expected behavior The precedence of configuration should be:

  1. The runnable's own config
  2. The suite's config
  3. The default value for the registered setting

For the command above:

JOB ID     : 9376b4739f2238c5cd97d828a2c484f856fb5697
JOB LOG    : /home/cleber/avocado/job-results/job-2024-08-12T06.01-9376b47/job.log
 (1/1) nothing-op: STARTED
 (1/1) nothing-op: PASS (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/cleber/avocado/job-results/job-2024-08-12T06.01-9376b47/results.html
JOB TIME   : 2.52 s

Current behavior Currently, the suite will overwrite a runnable's config with its own.

For the command above:

JOB ID     : 9376b4739f2238c5cd97d828a2c484f856fb5697
JOB LOG    : /home/cleber/avocado/job-results/job-2024-08-12T06.01-9376b47/job.log
 (1/1) noop: STARTED
 (1/1) noop: PASS (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/cleber/avocado/job-results/job-2024-08-12T06.01-9376b47/results.html
JOB TIME   : 2.52 s

System information (please complete the following information):