search

metan-ucw / runltp-ng

Minimalistic LTP testrunner
11 stars 16 forks source link

The new --exclude behaviour is weird #32

Closed grisu48 closed 3 years ago

grisu48 commented 3 years ago

Hi,

in the recent version of runltp-ng, the --exclude parameter does not behave as expected anymore. We expected that the exclude parameter behaves like "if the regex or part of it matches", while it appears that it changed to "the regex must match as a whole". More specifically, we expected the behaviour of --exclude to behave like egrep, so when part of the test run matches the given regex, it should be excluded. This triggered a false positive in our test runs, since msgstress04 was not excluded anymore.

My guess is that the changes introduced in https://github.com/metan-ucw/runltp-ng/commit/c0f37f2fa4507540f8bd47f23376fcb5914bf632 are responsible for this change of behaviour.

Example

We currently use the following:

perl -I runltp-ng runltp-ng/runltp-ng --run syscalls --exclude '_16\$|^(move_pages|msg(ctl|get|rcv|snd|stress)|sem(ctl|get|op|snd)|shm(at|ctl|dt|get)|fallocate06|syslog|inotify07|inotify08|epoll_wait02|getrusage04|writev03)' ...

This always excluded the test runs msgstress01-msgstress04. However since 5 days this given exclude command does not exclude msgstress anymore, despite given there (^(move_pages|msg(ctl|get|rcv|snd|stress)).

See https://openqa.suse.de/tests/6200693 for an example test run, where msgstress04 was scheduled to run, although it should have been excluded.

A fix from our side would be definitely possible, but IMHO the new behaviour of --exclude feel weird. IMHO it should be by default generous in matching the excluded test runs, while the user can make more explicit rules by a clever arrangement of ^ and $.

pevik commented 3 years ago

@wangli5665 FYI https://github.com/metan-ucw/runltp-ng/commit/13ff090eeb9ad69e02531649847c9adabe1c8e48 probably breaks --include as well.