p4lang / ptf

Packet Test Framework
Apache License 2.0
146 stars 99 forks source link

Guarantee that tests are loaded in stable order #169

Closed karolrom-intel closed 2 years ago

karolrom-intel commented 2 years ago

Default behavior of --test-order option is described as:

tests are run in the order in which they appear on command line

In case test_specs contains only group names, the order of tests within the single group is not defined and may depend on filesystem implementation.

To avoid this issue files/directories should be visited in a stable order.