Building on Windows (and Github CI)

mgautierfr commented 2 weeks ago

This issue is a tracking issue about the native compilation of our projects on Windows, including CI/CD on Github CI

Xapian compilation

This big (and most problematic) point is Xapian compilation. Xapian build system is based on autotools. It works well on Unix or Windows with tools like msys2 but we want to compile without msys2.

So a port of the build system has been made to Meson. This can be found in repository https://github.com/openzim/xapian-meson. This build system mostly work but have few issues:

Some unittest randomly failing. Those tests seems to write a lot of time on FS, so it may be some kind of race condition. They have been deactivated (https://github.com/openzim/xapian-meson/commit/fb7b5559ee4a2663c2572cca15a279004cbf20a6)
Dynamic library compilation fails because xapian doesn't define public api (exportdll). It uses kind of "export all" feature of autotools. Meson doesn't support that (https://github.com/mesonbuild/meson/issues/2132). For now, xapian build only static library (https://github.com/openzim/xapian-meson/commit/5b862a0ef5cfd6c3ff02800c3f4479830ba2bc75)
When compacting xapian database with flag COMPACT_FULLER, xapian crashes. So libzim has a compile option to not use this flag (https://github.com/openzim/libzim/pull/909/commits/bc65f301885619966f5304149fcfb6f8887c05f3)

Upstream issue is https://trac.xapian.org/ticket/832

Libmicrohttpd

We are using meson build system coming from wrapdb. It has a small bug about detection of a type length. A patch is applied (and present) in kiwix-build (https://github.com/kiwix/kiwix-build/pull/726). Patch has also be sent upstream and is under discussion (https://github.com/mesonbuild/wrapdb/pull/1625) Edit 2024-09-01 : Patch has been accepted upstream.

Libzim

Libzim compiles correctly (except COMPACT_FULLER thing) on Windows. Both static and dynamic. However, dynamic building generate a lot of warning as we have stl symbols in our class and we must re-export them. See d2d1f68 (#703)

Libkiwix

~~libkiwix was never tested on Windows. Now we can and we have discover some bugs. They are fixed in https://github.com/kiwix/libkiwix/pull/1113. Until then, test are deactivated on kiwix-build windows CI (https://github.com/kiwix/kiwix-build/pull/725/commits/04d0ba6a9212511cc0599d0f1d73c1b17c4b1255)~~ Tests are fixed on libkiwix.

Libkiwix build only statically as we don't export symbol (see "About dllexport symbols")

zim-tools

zimwriterfs is using libmagic to detect the mimetype of the file to add. libmagic is also using autotool as build system and we have to port it to meson as made for xapian (but should be simpler)

However, it not need as top priority as zimwriterfs is deactivated by zim-tools meson if we are compiling to/on Windows. https://github.com/kiwix/kiwix-build/pull/728 is building zim-tools on github windows CI.

Edit 2024-08-27: This is not only about libmagic. zimcheck also uses getopt to parse command line and I have strange error in metadata-test

Edit 2024-0901: zim-tools (without zimwriterfs) is compiled natively on Window using github CI

kiwix-tools

kiwix-tools uses getopt to parse command line. We currently have not build for it on Windows. We have several solutions:

Port kiwix-tools and use docoptcpp (as zim-tools) as it is already available in our toolchain (https://github.com/kiwix/kiwix-tools/issues/691)
getopt is a simple function, we can copy it in kiwix-tools and use this implementation as fallback (as libmagic is doing).
Find a project providing getopt on Windows as build it as dependency in kiwix-build.

Edit 2024-09-01: kiwix-tools has been ported to docoptcpp and is compiled natively on Window using github CI

kiwix-desktop

Dependencies of kiwix-desktop is build by kiwix-build.
kiwix-desktop CI is now using those dependencies to compile at each PR (https://github.com/kiwix/kiwix-desktop/pull/1184)
kiwix-build also publish nigthly of kiwix-desktop
~~Appveyor build is not removed. To remove once build in github CI is tested ok~~
~~kiwix-desktop is not signed.~~

Edit 2024-09-01: Kiwix compiles natively on Windows (github CI). It is published as nightly and release will be signed. Appveyor CI has been removed.

About dllexport symbols.

When compiling dynamically (dll) on Windows, we have to declare what symbols are exported in the dll (other symbols are private). This can be done either by using __declspec(dllexport) attribute in our headers or using a .def file which list exported symbols.

Libzim takes the first way. However, as said, it generates a lot of warning as we are declaring/using stl symbols in our class and we are not reexporting them. (See d2d1f68 (#703) and linked links for more information). The importance of those warning are not totally known. But they are annoying anyway. Note that using a .def file to export the same symbols would probably have the same effect.

Another solution is to export ALL symbols in the dll. The consequence of this is not totally known (at least by me). It seems that autotools and CMAKE have an option to generate a .def which contains all the symbols but meson have not (https://github.com/mesonbuild/meson/issues/2132).

I have ported such a script in python (https://gist.github.com/mgautierfr/11eb53379bf063f965614dcc9b6baef7) and propose it to meson to integrate it https://github.com/mesonbuild/meson/issues/2132

Quick testing shows that we could generate a dll and link to it.

Ping @rgaudin @kelson42 @veloman-yunkan

veloman-yunkan commented 2 weeks ago

Are we under a malware distribution attack?

mgautierfr commented 2 weeks ago

Are we under a malware distribution attack?

It seems so

kelson42 commented 1 week ago

@mgautierfr Thank you for the wrap-up, each of the remaining issue, should be put in a dedicated issue.

kiwix / kiwix-build