search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
345 stars 343 forks source link

Introduce "exec-runnables-recipe" resolver #6032

Closed clebergnu closed 1 month ago

clebergnu commented 2 months ago

This resolver is somewhat of a hybrid between the "exec-test" and the "runnables-recipe" resolvers.

It runs an executable, and attempts to read from its STDOUT content that will be treated as runnables-recipe JSON content. If that succeeds, the content will be returned as test resolutions. This is useful for executable tests or test generators that will output the tests dynamically.

Fixes: https://github.com/avocado-framework/avocado/issues/6009

clebergnu commented 1 month ago

Hi @clebergnu, I don't have many comments to the code itself, but I think we should discuss the overall behaviour of exec-runnables-recipe resolver.

IIUIC, with your solution we can combine exec-runnables-recipe and exec-test resolvers together, which IMO is not good behaviour, because we could end up in situations where tests are run during resolving phase. I would propose two possible solutions here.

True. Although I believe the solution we have for that is really the use of different suites. Users should be aware of the behavior, and control, through the use of different suites, if executables are meant to be run during the resolver phase or not.

  1. If --resolver-run-executables option is used all the other resolvers are disabled. This is easy solution, but we will lose the possibility of running multiple test types at once.

Right.

  1. --resolver-run-executables won't be a boolean, but it will be list where user can specify executables which should be directly run by exec-runnables-recipe resolver. All the other references will go through normal resolution process. I would prefer this kind of solution.

I think this is flexible, but adds "special" options and behavior to the overall resolver approach.

What do you think, about this problem.

Like I said before, I think users should pick the references themselves that should be executed for resolutions, and the ones that should not (that will be executed as tests instead).

I'll go ahead and better document the behavior and put a note about how users can not (or should not) combine those two different types of executables in a single suite.

richtja commented 1 month ago

@clebergnu, I just noticed that spell-check is failing. Can you please fix that? Thanks.

clebergnu commented 1 month ago

@clebergnu, I just noticed that spell-check is failing. Can you please fix that? Thanks.

Spell checking is happy now! Thanks for the review!