wwylele / teakra

DSi/3DS DSP emulator, disassembler, assembler, and tester
MIT License
76 stars 19 forks source link

Integrate accuracy test into CI #15

Closed liushuyu closed 5 years ago

liushuyu commented 5 years ago

In this PR, HW LLE accuracy test is included in CI checks.

Currently, due to a bug in Travis CI Windows agent, the test on that platform is disabled.


And I think there will be some work to do if you want to use Catch2 Unit Test Framework. It does not provide "skip" method, and you need to sub-class the Matcher to make it outputs the useful information.


This change is Reviewable

wwylele commented 5 years ago

Here are some problems I found:

liushuyu commented 5 years ago

Do I have to reupload the file every time cache is removed?

No, as you can see in CMakeLists.txt, I uploaded the file to a paid CDN service. They won't remove the file of course. I used the cache system in CI services because:

  1. Usually, their cache storage server is faster to be accessed in their containers
  2. To reduce my own cost

Could you also add the test to appveyor as well?

I will address this.

test_verifier currently doesn't return error code in case of test failure

That's fine, I can modify that small program to return a non-zero status if more than zero test fails.

does ctest output what test_verifier prints (at least on failure)? It would be nice to know which test case fails exactly.

Yes but only if test_verifier returns a non-zero status.

liushuyu commented 5 years ago

Issues addressed.