There are way to many possible combinations of file versions and features than I can possibly test. Instead of writing my own unit tests this MR uses existing designs on GitHub to check the parser against them.
repos.yaml contains a URL to the repositories and an error counter. The counter represents the number of errors that occurred while parsing the file. E.g. a symbol library file (*.OLB) contains 100 symbols where 43 fail and 57 pass. Then the 43 failures will be saved in repos.yaml. When the parser is modified at a later point in time and the test is run again we know that at most 43 fails are allowed because the other symbols passed in an earlier parser revision. I.e. a regression was introduced.
The generall workflow is shown in the following:
Add new repository to repos.yaml and preset the error counter to a very high value (99999)
Clone Git Repository
Generate unit tests that check the amount of failures as described above
Unit tests generate a log file containing the amount of failures in the current run
Check whether we have less failures than previously, then the parser was improved. As a result, reduce the error counter in repos.yaml accordingly
Steps 2 to 5 must be repeated for every change to the parser. Unfortunately this regression test will not run in the CI because it takes roughly 45 min to run those tests where a log file of 3 GB is created. Disabling the log file creation is left for a future todo.
There are way to many possible combinations of file versions and features than I can possibly test. Instead of writing my own unit tests this MR uses existing designs on GitHub to check the parser against them.
repos.yamlcontains a URL to the repositories and an error counter. The counter represents the number of errors that occurred while parsing the file. E.g. a symbol library file (*.OLB) contains 100 symbols where 43 fail and 57 pass. Then the 43 failures will be saved inrepos.yaml. When the parser is modified at a later point in time and the test is run again we know that at most 43 fails are allowed because the other symbols passed in an earlier parser revision. I.e. a regression was introduced.The generall workflow is shown in the following:
repos.yamland preset the error counter to a very high value (99999)repos.yamlaccordinglySteps 2 to 5 must be repeated for every change to the parser. Unfortunately this regression test will not run in the CI because it takes roughly 45 min to run those tests where a log file of 3 GB is created. Disabling the log file creation is left for a future todo.