Fix non-deterministic test order - Githubissues

rapid7 / recog

Pattern recognition for hosts, services, and content

Other

671 stars 199 forks source link

Fix non-deterministic test order #592

Closed adfoster-r7 closed 1 year ago

adfoster-r7 commented 1 year ago

Description

Fix non-deterministic test order when running tests. A test failure on CI outputs an error with the reproducible test steps:

Failed examples:

rspec './spec/lib/fingerprint_self_test_spec.rb[1:29:120:3]' # Recog::DB#http_cookies.xml (?-mix:^(__Host-)?(nc_sameSiteCookiestrict|nc_sameSiteCookielax|oc_sessionPassphrase)=) uses capturing regular expressions properly

However - the test may not actually for a developer's local dev env, as Dir is used without an explicit sort:

Overview. 'DirandDir. glob(…)` do not make any guarantees about the order in which files are returned. The final order is determined by the operating system and file system.

Motivation and Context

Developers want consistency between CI and local testing env

How Has This Been Tested?

CI Passes

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] I have updated the documentation accordingly (or changes are not required).
[x] I have added tests to cover my changes (or new tests are not required).
[x] All new and existing tests passed.