Run test suite on Windows

[alerque]

As part of working up to a better release process (#588) and now that we have a bit more robust test suite (#587, Docs) it occurred to me that we should probably be running the actual test suite on Windows, not just making sure SILE compiles. This should probably be done before a Windows distribution is included (#410).

This might be one for @harrysummer or somebody who actually has a Windows box to test on to look into. I have no idea what make test is going to return on Windows right now. Do I need to adapt the test suite execution for Cmake?

[harrysummer]

Let me try to integrate test suite in the the CI build system for Windows.

[alerque]

I'd be happy to talk about the make/cmake situation here or in chat, right now it seems like we have two fairly different build systems. I've been working on the *nix side both baking much smarter logic into the test suite but also (pending in #648) changing up the way Lua libraries are handled and actually getting ready to bundle them in releases more similar to what (I think) you did on the Windows side. It seems like there might be a way to unify those two paths a little bit more now.

[harrysummer]

I agree that a unified building script would be cool. And there is always a debate between using make vs cmake. Personally, I would vote for cmake. Here is why:

We know make is the Linux gold standard. The rule-based script is straight-forward and extensible. But we should also see the growth of cmake -- the target-based system is intuitive and platform-independent. From what I see in the open-source community, more and more projects are adopting CMake as their build script (For some that don't, non-official CMake script is most likely to be there already). So, from the community support and ecosystem point-of-view, CMake is definitely the winner. And will benefit open-source projects in the long run.

For the current stage of SILE, specifically. We are starting from make and we did a lot of logic in there. What I suggest is that we keep the current make build system for Linux and let me continue refining the CMake script on Windows. When CMake is mature enough, we could start porting it back to Linux. And somebody with an Apple device may also port it into MacOS. Currently the CMake script still has some issues yet to solve including the test suite problem this issue is trying to address. After these issues are solved, we could move on to explore the possibility of integrating CMake into other platforms.

[alerque]

My own experience with cmake hasn't been so ... affable. I do know make is even more unfriendly out of the gate but in my case it's the devil I know and –in spite of it's sometimes absurd idiosyncrasies– there are reasons it's still the *nix gold standard.

I'm not adverse to being proven wrong and as long as we don't try to replace make before the alternative is fully baked, then I'll even help out in whatever way I can making the cmake porting process as easy as I can.

[harrysummer]

Got to make test working in Windows CI build using CTest framework in https://github.com/simoncozens/sile/pull/650. However, many test cases have non-parity results. For example, for the result of tests\bug-162.sil, the diff between expected and actual:

--- tests/bug-162.expected      2019-03-28 17:23:35.031412200 -0700
+++ build/tests/bug-162.actual  2019-06-21 13:08:13.018215900 -0700
@@ -8,17 +8,17 @@
 T      79 76 81 72 86  (lines)
 Mx     85.2098
 T      69 72 79 82 90  (below)
-Mx     112.3704
+Mx     112.2191
 T      86 75 82 88 79 71       (should)
-Mx     142.6512
+Mx     142.4999
 T      72 81 71        (end)
-Mx     160.6516
+Mx     160.5004
 T      76 81   (in)
-Mx     171.5428
+Mx     171.3915
 T      68      (a)
-Mx     178.7962
+Mx     178.6449
 T      83 72 85 76 82 71       (period)
-Mx     205.6126
+Mx     205.4613
 T      17      (.)
 Mx     40.9764
 My     49.3321

[harrysummer]

I just notice that in the Linux regression test, there are some tests that are known to fail or missing expect files. Maybe I will firstly rule them out in CMake script. However, we do have a lot of test cases whose results are not consistent between Linux and Windows.

[harrysummer]

Looked at https://github.com/simoncozens/sile/issues/639, https://github.com/simoncozens/sile/issues/319, and some discussion on Gitter. But this might not be a problem with Harfbuzz because the Windows build uses the same version of Harfbuzz as one of the Linux build.

Linux build Harfbuzz version: https://github.com/simoncozens/sile/blob/c699450af691f35ef3269e17b02dc63d137b53db/.travis.yml#L30

Windows build Harfbuzz version: https://github.com/simoncozens/sile/blob/eac1492f7eabc58a16383eb21d3bc27f58f2dd0f/CMakeLists.txt#L46

Or maybe Harfbuzz does something different on Windows?

@alerque @simoncozens Have any idea?

[harrysummer]

I tried to run this: https://github.com/simoncozens/sile/issues/639#issuecomment-502029027 on my Windows. The result is Height = 0

[simoncozens]

OK, that's bad! Did you confirm that the file ".fonts/LibertinusSerif-Regular.otf" exists?

(Edited) Oh, wait, the assert will catch that. So maybe it's a subshaper issue. Can you run

hb-shape --shapers=ot --show-extents .fonts/LibertinusSerif-Regular.otf 'a'

and also

hb-shape --list-shapers

[harrysummer]

Thanks, let me try to build glib so that I can build harfbuzz utils.

[harrysummer]

Here are the results:

D:\Repos\sile>hb-shape --shapers=ot --show-extents .fonts/LibertinusSerif-Regular.otf 'a'
[gid8=0+190<53,644,86,-218>|gid66=1+457<36,439,419,-449>|gid8=2+190<53,644,86,-218>]

D:\Repos\sile>hb-shape --list-shapers
ot
fallback

I also tried to run the same script on ArchLinux with latest harfbuzz. The results are:

$ hb-shape --shapers=ot --show-extents .fonts/LibertinusSerif-Regular.otf 'a'
[a=0+457<36,439,419,-449>]

$ hb-shape --list-shapers
graphite2
ot
fallback

Looks like the Windows version of hb-shape found more than one glyph for 'a'?

---

Edit: Oh, you can't use single quote on Windows. So:

D:\Repos\sile>hb-shape --shapers=ot --show-extents .fonts/LibertinusSerif-Regular.otf "a"
[gid66=0+457<36,439,419,-449>]

The true result is consistent with Linux, although I don't know why it is gid66 instead of a -- this might indicate a different execution path on Windows.

[khaledhosny]

The font functions used might be more relevant than the shaper here, try testing with --font-funcs=ot.