Adding functional tests to the cmake -DBUILD_TESTS=ON configuration in libxlsxwriter

[jmcnamara]

The cmake -DBUILD_TESTS=ON configuration builds the libxlsxwriter unit tests but to date it hasn't built the functional tests.

The libxlsxwriter functional tests use Python pytest to compile complete C programs (like this), runs them, and compare the outputs file by file and element by element with the XML in files generate by Excel. Currently there are around 800 of these tests and they ensure a high level of fidelity between the files produced by libxlsxwriter and Excel.

I have been enabling GitHub actions to improve the CI since I no longer have access to travis and I made a pass at enabling these functional tests.

They require Python and Pytest. Pytest can be installed like this:

sudo apt-get install -y python-pytest

# Or this:
pip install pytest

Then you can run the entire test suite like this:

cd libxlsxwriter
mkdir build
cd build
cmake .. -DBUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build . --config Release
ctest -C Release -V

There are some caveats:

• I need to add support for testing some of the compilation variants with Cmake/Ctest (they are already in the "make" tests).
• The Cmake code is probably a bit clunky, comments/suggestions welcome: https://github.com/jmcnamara/libxlsxwriter/blob/main/CMakeLists.txt#L356
• The check for pytest doesn't work. I've tried a lot of find_package() and find_program() variations and I can't get it to work. Suggestions welcome here as well.
• I need to add Windows support/CI tests.

@sjmulder @Alexhuszagh @remicollet @manisandro and any other packagers/porters/cmake users: could you have a look if you get a chance and let me know if you have any feedback.

[jmcnamara]

The check for pytest doesn't work. I've tried a lot of find_package() and find_program() variations and I can't get it to work. Suggestions welcome here as well.

I fixed this and tested it with a clean Ubuntu install without and with pytest.

[jmcnamara]

I need to add support for testing some of the compilation variants with Cmake/Ctest (they are already in the "make" tests).

I've added this now as well for most/all Libxlsxwriter Cmake configuration options:

jobs:
  build:
    name:
      Cmake
    strategy:
      matrix:
        cc:          [gcc, clang]
        cmake_flags: ["",
                      "-DBUILD_EXAMPLES=ON       -DBUILD_TESTS=OFF",
                      "-DUSE_DTOA_LIBRARY=ON     -DBUILD_TESTS=ON",
                      "-DUSE_FMEMOPEN=ON         -DBUILD_TESTS=ON",
                      "-DUSE_NO_MD5=ON           -DBUILD_TESTS=ON",
                      "-DUSE_OPENSSL_MD5=ON      -DBUILD_TESTS=ON",
                      "-DUSE_STANDARD_TMPFILE=ON -DBUILD_TESTS=ON",
                      "-DUSE_SYSTEM_MINIZIP=ON   -DBUILD_TESTS=ON",
                      "-DUSE_SYSTEM_MINIZIP=ON   -DUSE_OPENSSL_MD5=ON -DBUILD_TESTS=ON"]
    runs-on: ubuntu-latest

[manisandro]

I gave it a quick spin on Fedora and everything seemed to work fine, incl pytest detection.

$ cmake -DUSE_SYSTEM_MINIZIP=ON -DBUILD_TESTS=ON
[...]
$ make
[...]
$ ctest
Test project /home/sandro/Documents/Devel/Fedora/libxlsxwriter/libxlsxwriter-main/build
    Start 1: xlsxwriter_unit
1/2 Test #1: xlsxwriter_unit ..................   Passed    0.03 sec
    Start 2: functional
2/2 Test #2: functional .......................   Passed   10.92 sec

100% tests passed, 0 tests failed out of 2

[jmcnamara]

@manisandro Thanks for testing.

Note, in case anyone hits it, there is currently an issue with building the examples and tests together. I'm working on that. Fixed it.

[jmcnamara]

I've added a Windows Cmake build to the CI. It is quite slow to build OpenSSL and the test cases (~20 mins) but it may catch some issues. There are some minors warning from MSVC that other compilers don't raise so I need to look at them too.

The Windows builds are:

        cmake_flags: ["-DBUILD_EXAMPLES=ON       -DBUILD_TESTS=ON",
                      "-DUSE_DTOA_LIBRARY=ON     -DBUILD_TESTS=ON",
                      "-DUSE_OPENSSL_MD5=ON      -DBUILD_TESTS=ON",
                      "-DUSE_SYSTEM_MINIZIP=ON   -DBUILD_TESTS=ON",
                      "-DUSE_SYSTEM_MINIZIP=ON   -DUSE_OPENSSL_MD5=ON -DBUILD_TESTS=ON"]

I also had some issues with Ctest that I had to work around.

I'm going to close this issue now since this is the majority of the code tests that I want to put in the CI. I may add an automated code formatting check as well but that is non-critical.

[jmcnamara]

For the package maintainers CCed on this issue.

I'm updating the libxlsxwriter installation instructions. If you have any additional package manager instructions that I could add please let me know.

[jmcnamara]

For what it is worth I have restructured the Getting Started Guide to group more instructions in terms of OSes and to move cmake onto an even par with make in the instructions.

https://libxlsxwriter.github.io/getting_started.html