search

usnistgov / iheos-toolkit2

XDS Toolkit
https://ihexds.nist.gov
46 stars 26 forks source link

Feature Requested: Sorted Test List #554

Closed stl-steve-moore closed 1 year ago

stl-steve-moore commented 2 years ago

I would assign this to me. I just need a place to track it and see if you agree with the request and/or approach.

Tests for an actor are displayed in alphabetical order based on their name (taken from the name on the folder). I have created some tests with prefixes so I have them displayed in the order that I want the to be executed, but that just takes up extra space.

Test Steps are ordered by the index.idx file, but that is not really appropriate here. You are using collections.txt inside each test case and do not want another mechanism where you list "all tests for XCA with option ..." in some order. That will likely lead to maintenance issue.

I am suggesting something that will also lead to maintenance issues, but it is more in line with your collections.txt mechanism to identify the tests for a profile/actor/option.

Suggest using yet another file at the test level, something like display_order.txt. This could be a set of key/value pairs where the keys are the text values found in display_order.txt. When the software finds all of the tests for one of those [ xca(rg) ], it also looks in display_order.txt to get a hint about order. Values are numbered from 1 to ... and you expect both gaps and duplicates. I would not worry about the duplicates nor the gaps. As long as there is some order, the author can work to get it right.

stl-steve-moore commented 1 year ago

Code was added in branch feature/sort-tests and merged with #560