This issue is about improvement ideas to the way we test code:
### Tasks
- [ ] https://github.com/CGAL/cgal/issues/8165
- [ ] https://github.com/CGAL/cgal/issues/8166
- [ ] https://github.com/CGAL/cgal/issues/8167
- [x] The "y" square at the top of the "y" column is missing the green color (fixed by #7314)
- [x] Distinguish between error and time-out
- [x] Show run time for each executable
- [ ] https://github.com/CGAL/cgal/issues/8162
- [ ] https://github.com/CGAL/cgal/issues/8425
- [ ] https://github.com/CGAL/cgal/issues/8424
- [ ] https://github.com/CGAL/cgal/issues/8423
- [x] CAN NOT BE IMPLEMENTED. Compare runtimes of each executable over consecutive runtime testsuites and report large swings (~thus creating some kind of benchmarking tests). Could be a "S" letter (for Speed), yellow (and put warnings to orange).
- [ ] https://github.com/CGAL/cgal/issues/8164
- [ ] https://github.com/CGAL/cgal/issues/8426
[x] The page test suite result page could use a tool to compare between two chosen test suite runs, or at least a comparison to the fixed test suite that runs the previous release without anything. It would show some kind of (hopefully much smaller) diff between the two results. I know it's explained how to do it in command line somewhere, but a nice GUI on top on the website would be better.
[x] [Outdated but same-ish problem with Ctest] When testing branch packages with create_cgal_test_with_cmake, the compilation output is dumped in the console, which is mostly unreadable and difficult to search for warnings. I guess one could dump the output into a file and search the file, but maybe the "successful" / "error" message on the left of the program in error.txt could show if a warning popped up? The grep to detect the warnings can be picked from collect_cgal_testresults_from_cmake. Just use -Werror
This issue is about improvement ideas to the way we test code:
diff
between the two results. I know it's explained how to do it in command line somewhere, but a nice GUI on top on the website would be better.[Outdated but same-ish problem with Ctest] When testing branch packages withJust usecreate_cgal_test_with_cmake
, the compilation output is dumped in the console, which is mostly unreadable and difficult to search for warnings. I guess one could dump the output into a file and search the file, but maybe the "successful" / "error" message on the left of the program inerror.txt
could show if a warning popped up? Thegrep
to detect the warnings can be picked fromcollect_cgal_testresults_from_cmake
.-Werror