Override test duration from plan

teemtee / tmt

Test Management Tool

MIT License

80 stars 122 forks source link

Override test duration from plan #1843

Open pellecchialuigi opened 1 year ago

pellecchialuigi commented 1 year ago

As a tester, I may need to modify some libraries that the test uses. Those instrumentations can significantly change the test duration and I need to be able to run those tests in that scenario without requesting changes in the test fmf file. A possible solution could be to override the duration using a mathematical operation. example: duration: *3 example: duration: +1h

lukaszachy commented 1 year ago

Outcome of the discussion on the hacking session:

Create new attribute which will allow to change the duration: either to set explicit value or mathematical operation on existing value ( +1s, +1m, +1h, *1.2 )

Attribute could be used in discover phases where it will affect tests gathered in that phase as well as in execute step where it will affect all discovered tests.

lukaszachy commented 1 year ago

Use case we should cover: One arch is slower than the others so:

adjust:
 - when: arch == x86_64
   duration: '*2'

lukaszachy commented 1 year ago

Lets' aim for next release.

psss commented 1 year ago

Has the essential part of the use case been covered by #1972? Currently duration allows to provide multiple values:

# Combination & repetition of time suffixes (total 4h 2m 3s)
duration: 1h 3h 2m 3

So it's possible to use context adjust to extend it with a custom value. Is the multiplication support important enough to dive into this now?

lukaszachy commented 1 year ago

Unfortunatelly we are still missing way how to modify duration from the PLAN, which is necessary when

one cannot modify test repo
plan changes duration significantly (e.g. experiment on very slow machine, or running with profiling, custom LD_PRELOAD....)

psss commented 1 year ago

Ah, I see, the plan part is here essential.

happz commented 11 months ago

@lukaszachy is this issue going to land in 1.29? I'd propose postponing once more to 1.30.

lukaszachy commented 11 months ago

Started https://github.com/teemtee/tmt/discussions/2430

lukaszachy commented 5 months ago

Status comment:

[x] #2845.
[x] fmf feature to call adjust with explicit set of rule - https://github.com/teemtee/fmf/pull/230
[ ] tmt discover attribute to store explicit rules - #2865
[x] fmf merge method to "sed" values - https://github.com/teemtee/fmf/pull/234

Not sure if 1.33 is feasible as the third point will require fmf release to happen first.