win10 native MSYS2 build make package error

[Melonbob]

Ran into error running cmake --build . --target package (@matty0ung that is confirmed a good command instead of default mingw32-make package)

From the log NSISOutput.log

!include: could not find: "Locate.nsh"

Error in script "C:/msys64/usr/local/build/_CPack_Packages/win64/NSIS/project.nsi" on line 1008 -- aborting creation process

[matty0ung]

It needs some plug-ins:

• Download https://nsis.sourceforge.io/mediawiki/images/a/af/Locate.zip and https://nsis.sourceforge.io/mediawiki/images/7/76/Nsislog.zip to a temporary directory.
• Unpack Locate.zip into, say, Locate subdirectory.
• Unpack Nsislog.zip into, say, Nsislog subdirectory.
• Copy Locate/Include/Locate.nsh to /mingw32/share/nsis/Include/
• Copy Locate/Plugin/locate.dll and Nsislog/plugin/nsislog.dll to /mingw32/share/nsis/Plugins/ansi/

[Melonbob]

Got passed that one only by putting Locate.nsh in /mingw64/share/nsis/Include/ as well and DLLs as above.

Next one:

NSISOutput.log

Plugin not found, cannot call locate::_Open

Error in macro locate::Open on macroline 1

Error in script "C:/msys64/usr/local/build/_CPack_Packages/win64/NSIS/project.nsi" on line 1133 -- aborting creation process

[matty0ung]

You may find that you need to build 32-bit rather than 64-bit to get things to work. Launch the MSYS2 MinGW 32-bit terminal instead of the MSYS2 MinGW 64-bit one. I don't know the details, but I think NSIS is itself 32-bit and that, although it can install 64-bit applications, there's a bit more involved in making that happen.

In the current build system, where CMake drives CPack drives NSIS, there are limits to what we can control (or at least work out what to control). Perhaps in future if we're driving NSIS directly it will be easier to fine-tune it.

[Melonbob]

Ok, installed all mingw-w64-i686- tool chain stuff and that did not complain even once. I'll post list of stuff here and give the wiki instructions an edit.

[Melonbob]

Packaged version crashes App upon export of records.

[matty0ung]

What about if you do a direct local install?

cmake --build . --target install
brewtarget

If that works, then we know the problem is with the packaging. If it doesn't then it would point us to looking elsewhere.

[Melonbob]

Indeed, direct local installs work just fine with both MSYS2 MINGW64 on win10 and win11 and MSYS2 MINGW32 on win11 builds with admin-privileged terminals (I didn't try the 32bit MSYS2 MINGW32 toolchain on win10). However only MSYS2 MINGW32 was able to complete the assembly of the .exe package (cmake --build . --target package ), and once installed, crashed the app when trying to export hop, yeast or recipe records. The direct local installs do not crash the app.

[matty0ung]

OK, that's good to know. Thanks for all your efforts here.

So, my best guess at the moment is that we're missing something vital from the package. I imagine, somewhere soon after https://github.com/Brewtarget/brewtarget/blob/e92a594ec7223d009a75b450361ed499a24741a1/CMakeLists.txt#L1289 we need to do something extra related to Xerces or Xalan libraries. OR, we have to find a way to tell CMake that we just want to statically link those libraries (and their dependencies) so that we don't have to ship them.

You'll see from looking at the code both the advantage and the disadvantage of using CPack. The plus is that it does a lot of boilerplate work for you so, in theory, you have very little work to do to create installers and packages across all 3 platforms (Windows, Mac and Linux). The minus is that it's easy to hit edge cases where it seems hard to get the end result that you want (see endless comments in the Mac section for example) and all the wizardry going on under the hood makes it a bit tricky to debug.

[Melonbob]

I'm trying to get Visual Studio builds going to see if its any different. My win10 has VS2017 and lots of cygwin and other downloaded binaries in the paths, so I don't know how reproducible anything coming from it could be. The win11 has virgin installs of these tool chains but VS2022---argh!--- is not as friendly. (Microsoft stuff and I have had a torrid relationship for years.)

[Melonbob]

Didn't get very far this week with a VS2022 build kit in just trying to get xerces-c and xalan-c built from source. At one point I had them both configured with cmake, xerces-c built and installed, but the more ran down build issues with xalan-c, the more new errors popped up. MS stuff...

I noticed the MSYS2 MINGW32 brewtarget.exe of the packaged install was 7748k while the direct install of it was 149420k. All DLLs were the exactly the same and in size as well.

[Melonbob]

I just looked at a trial of PE Explorer. In the Dependency Scanner, when I hit xerces-c-3-2.dll it showed version info not available. This was one of the more new errors I mentioned. Here is the report: brewtargetPKG.depend.txt This was for the packed intall version.

For the directly installed version, compare this report: brewtargetDIR.depend.txt

I will put them in a side by side spreadsheet and analyze ...

[Melonbob]

Disappointing it show anything differential but here it is. MSYS2-MINGW32-PKG-DIR-Dependencies.xlsx

[matty0ung]

That seems like a promising line of enquiry. The fact that the thing works on the build machine but not when it's installed from package makes me think that we're missing something, perhaps a DLL, from the package.

If this output gives us clues about which DLL or DLLs that might be, a first thing to try might be manually copying additional DLLs into C:\PROGRAM FILES (X86)\BREWTARGET-3.0.6A32\BIN\ and see if that stops the thing crashing. (I don't have an in-depth knowledge of the DLL loading process on Windows. I sort of assumed that you always get the helpful pop-up window when a program asks to load a DLL that the system doesn't find. But maybe if it's a DLL asking to load another DLL it's different.)

[matty0ung]

I am, by the way, attempting to come at things from the other direction. In my new experimental build system, I have a couple of scripts that build the Windows package manually (or rather invoke NSIS manually) rather than doing things via CPack (which is more automated but gives you less control). If I can get that building, that will give me a list of the DLLs I needed to include to get it to work. We can then try adding that list to the CMake scripts we currently use for packaging.

[matty0ung]

So, I've now reached a very interesting point. I have two build systems running side-by-side, old and new. Installing the packages from them gives different results. On the old (ie current CMake/CPack) build system, I can reproduce the problem - exporting to XML causes a crash without much useful diagnostics. On the new (ie Meson + Python) build system, I can't reproduce the problem - exporting to XML and importing from XML seems to work fine.

Here's what gets installed in the bin directory from the installer generated by the "old" build system:

# ls -l /C/Program\ Files\ \(x86\)/Brewken-0.1.0/bin/
total 97984
-rwxr-xr-x 1 Matt None  7498668 Jan  5 20:07 Qt5Core.dll
-rwxr-xr-x 1 Matt None  9958502 Jan  5 20:07 Qt5Gui.dll
-rwxr-xr-x 1 Matt None  1552564 Jan  6 12:22 Qt5Multimedia.dll
-rwxr-xr-x 1 Matt None  2601183 Jan  5 20:07 Qt5Network.dll
-rwxr-xr-x 1 Matt None   635828 Jan  5 20:07 Qt5PrintSupport.dll
-rwxr-xr-x 1 Matt None   402426 Jan  5 20:07 Qt5Sql.dll
-rwxr-xr-x 1 Matt None   586457 Jan  6 11:39 Qt5Svg.dll
-rwxr-xr-x 1 Matt None  9298041 Jan  5 20:07 Qt5Widgets.dll
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 audio
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 bearer
-rwxr-xr-x 1 Matt None  9817102 Jan 30 17:16 brewken.exe
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 iconengines
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 imageformats
-rwxr-xr-x 1 Matt None   144123 Jun 11  2022 libbrotlicommon.dll
-rwxr-xr-x 1 Matt None    55152 Jun 11  2022 libbrotlidec.dll
-rwxr-xr-x 1 Matt None   629591 Jun 11  2022 libbrotlienc.dll
-rwxr-xr-x 1 Matt None   105784 Dec 22  2020 libbz2-1.dll
-rwxr-xr-x 1 Matt None    83390 Aug 19 15:55 libdouble-conversion.dll
-rwxr-xr-x 1 Matt None   800273 May  1  2022 libfreetype-6.dll
-rwxr-xr-x 1 Matt None   145607 Dec 27 06:26 libgcc_s_dw2-1.dll
-rwxr-xr-x 1 Matt None  1496724 Dec 22 21:43 libglib-2.0-0.dll
-rwxr-xr-x 1 Matt None   161663 Sep 10  2020 libgraphite2.dll
-rwxr-xr-x 1 Matt None  1218019 Dec 17 16:26 libharfbuzz-0.dll
-rwxr-xr-x 1 Matt None    80510 Dec 17 16:26 libharfbuzz-gobject-0.dll
-rwxr-xr-x 1 Matt None    19632 Dec 17 16:26 libharfbuzz-icu-0.dll
-rwxr-xr-x 1 Matt None  1146664 Dec 17 16:26 libharfbuzz-subset-0.dll
-rwxr-xr-x 1 Matt None  1120784 Jun  9  2022 libiconv-2.dll
-rwxr-xr-x 1 Matt None 31266523 Oct 24 06:43 libicudt72.dll
-rwxr-xr-x 1 Matt None  3321641 Oct 24 06:43 libicuin72.dll
-rwxr-xr-x 1 Matt None    65084 Oct 24 06:43 libicuio72.dll
-rwxr-xr-x 1 Matt None   103641 Oct 24 06:43 libicutest72.dll
-rwxr-xr-x 1 Matt None   287231 Oct 24 06:43 libicutu72.dll
-rwxr-xr-x 1 Matt None  1950581 Oct 24 06:43 libicuuc72.dll
-rwxr-xr-x 1 Matt None   147235 Jan 12  2022 libintl-8.dll
-rwxr-xr-x 1 Matt None   108495 Jun 27  2022 libmd4c-html.dll
-rwxr-xr-x 1 Matt None    87452 Jun 27  2022 libmd4c.dll
-rwxr-xr-x 1 Matt None   359466 Dec 17 17:59 libpcre2-16-0.dll
-rwxr-xr-x 1 Matt None   348714 Dec 17 17:59 libpcre2-32-0.dll
-rwxr-xr-x 1 Matt None   398922 Dec 17 17:59 libpcre2-8-0.dll
-rwxr-xr-x 1 Matt None   268666 Nov 21 22:15 libpng16-16.dll
-rwxr-xr-x 1 Matt None  1413815 Jan  1 11:39 libsqlite3-0.dll
-rwxr-xr-x 1 Matt None  2224520 Dec 27 06:26 libstdc++-6.dll
-rwxr-xr-x 1 Matt None    71838 Jan  8 12:41 libwinpthread-1.dll
-rwxr-xr-x 1 Matt None  3706786 Oct 24 10:29 libxalan-c.dll
-rwxr-xr-x 1 Matt None    40851 Oct 24 10:29 libxalanMsg.dll
-rwxr-xr-x 1 Matt None  3544657 Dec  3 06:27 libxerces-c-3-2.dll
-rwxr-xr-x 1 Matt None   823683 Mar  3  2022 libzstd.dll
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 mediaservice
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 platforms
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 playlistformats
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 printsupport
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 sqldrivers
drwxr-xr-x 1 Matt None        0 Jan 30 17:20 styles
-rwxr-xr-x 1 Matt None   127345 Nov 18 15:44 zlib1.dll
#

And here's what gets installed from the installer generated by the new build system:

# ls -l /C/Program\ Files\ \(x86\)/Brewken-0.1.0/bin/
total 326900
-rwxr-xr-x 1 Matt None   7498668 Jan  5 20:07 Qt5Core.dll
-rwxr-xr-x 1 Matt None   9958502 Jan  5 20:07 Qt5Gui.dll
-rwxr-xr-x 1 Matt None   1552564 Jan  6 12:22 Qt5Multimedia.dll
-rwxr-xr-x 1 Matt None   2601183 Jan  5 20:07 Qt5Network.dll
-rwxr-xr-x 1 Matt None    635828 Jan  5 20:07 Qt5PrintSupport.dll
-rwxr-xr-x 1 Matt None    402426 Jan  5 20:07 Qt5Sql.dll
-rwxr-xr-x 1 Matt None    586457 Jan  6 11:39 Qt5Svg.dll
-rwxr-xr-x 1 Matt None   9298041 Jan  5 20:07 Qt5Widgets.dll
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 audio
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 bearer
-rwxr-xr-x 1 Matt None 246042644 Jan 30 17:03 brewken.exe
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 iconengines
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 imageformats
-rwxr-xr-x 1 Matt None    144123 Jun 11  2022 libbrotlicommon.dll
-rwxr-xr-x 1 Matt None     55152 Jun 11  2022 libbrotlidec.dll
-rwxr-xr-x 1 Matt None    629591 Jun 11  2022 libbrotlienc.dll
-rwxr-xr-x 1 Matt None    105784 Dec 22  2020 libbz2-1.dll
-rwxr-xr-x 1 Matt None     83390 Aug 19 15:55 libdouble-conversion.dll
-rwxr-xr-x 1 Matt None    800273 May  1  2022 libfreetype-6.dll
-rwxr-xr-x 1 Matt None    145607 Dec 27 06:26 libgcc_s_dw2-1.dll
-rwxr-xr-x 1 Matt None   1496724 Dec 22 21:43 libglib-2.0-0.dll
-rwxr-xr-x 1 Matt None    161663 Sep 10  2020 libgraphite2.dll
-rwxr-xr-x 1 Matt None   1218019 Dec 17 16:26 libharfbuzz-0.dll
-rwxr-xr-x 1 Matt None   1120784 Jun  9  2022 libiconv-2.dll
-rwxr-xr-x 1 Matt None  31266523 Oct 24 06:43 libicudt72.dll
-rwxr-xr-x 1 Matt None   3321641 Oct 24 06:43 libicuin72.dll
-rwxr-xr-x 1 Matt None   1950581 Oct 24 06:43 libicuuc72.dll
-rwxr-xr-x 1 Matt None    147235 Jan 12  2022 libintl-8.dll
-rwxr-xr-x 1 Matt None     87452 Jun 27  2022 libmd4c.dll
-rwxr-xr-x 1 Matt None    359466 Dec 17 17:59 libpcre2-16-0.dll
-rwxr-xr-x 1 Matt None    348714 Dec 17 17:59 libpcre2-32-0.dll
-rwxr-xr-x 1 Matt None    398922 Dec 17 17:59 libpcre2-8-0.dll
-rwxr-xr-x 1 Matt None    268666 Nov 21 22:15 libpng16-16.dll
-rwxr-xr-x 1 Matt None   1413815 Jan  1 11:39 libsqlite3-0.dll
-rwxr-xr-x 1 Matt None   2224520 Dec 27 06:26 libstdc++-6.dll
-rwxr-xr-x 1 Matt None     71838 Jan  8 12:41 libwinpthread-1.dll
-rwxr-xr-x 1 Matt None   3706786 Oct 24 10:29 libxalan-c.dll
-rwxr-xr-x 1 Matt None     40851 Oct 24 10:29 libxalanMsg.dll
-rwxr-xr-x 1 Matt None   3544657 Dec  3 06:27 libxerces-c-3-2.dll
-rwxr-xr-x 1 Matt None    823683 Mar  3  2022 libzstd.dll
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 mediaservice
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 platforms
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 playlistformats
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 printsupport
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 sqldrivers
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 styles
drwxr-xr-x 1 Matt None         0 Jan 30 17:25 translations
-rwxr-xr-x 1 Matt None    127345 Nov 18 15:44 zlib1.dll

(I'm doing the work in Brewken, but the "old" build system and most of the code is identical with Brewtarget.)

This is after I added all the "extra" DLLs to the old build system. So, even though we're now installing pretty much the same DLLs between the two systems (in fact even a couple of extra ones for the old system), we're still seeing the different behaviour. To me, from the listings above it looks like the biggest difference is now the size of the main executable: 9,817,102 bytes for the old build and 246,042,644 bytes for the new one. I'm guessing that somehow on the "new" build we're managing to link more in statically than on the "old" one.

[matty0ung]

I'd like to find a fix that patches up the existing build system. It feels wrong to completely change how we do build and packaging for a point release (ie 3.0.5 -> 3.0.6). In theory, I can just see what build and link flags are being emitted by both systems and then reverse-engineer the CMake settings to make it do the same as the Meson ones. In practice, I don't know whether it will be that simple. Let's see.

What I probably will do in parallel is port the new build system across to Brewtarget. This is not as scary as it might sound. The new build system does not modify the old one and can happily run along side it, not least because all the output goes in a different directory (mbuild instead of build). This will make it easier for anyone else who is interested to poke around at both builds and see what they can see.

[Melonbob]

I came across this bug report for MSYS2 MINGW32 builds with -static-libgcc links. Locate was built with it. See Locate/Source/makefile in https://nsis.sourceforge.io/mediawiki/images/a/af/Locate.zip

[matty0ung]

I came across this bug report for MSYS2 MINGW32 builds with -static-libgcc links. Locate was built with it. See Locate/Source/makefile in https://nsis.sourceforge.io/mediawiki/images/a/af/Locate.zip

Oh, that's interesting. Will have to check the compiler flags. I would think it would not be the cause of our problem, but it might perhaps be why we aren't getting great diagnostics from the crash.

[Melonbob]

https://gcc.gnu.org/onlinedocs/gcc-12.2.0/gcc/Link-Options.html#Link-Options under -static-libgcc

There are several situations in which an application should use the shared libgcc instead of the static version. The most common of these is when the application wishes to throw and catch exceptions across different shared libraries. In that case, each of the libraries as well as the application itself should use the shared libgcc.

[Melonbob]

Don't know why but line 37 is the only one in build/CPackConfig.cmake without a full path. All three MINGW64, MINGW32 and UCRT64 environments are like this. The build/CMakeCache.txt has it defined with a full path. It comes from cmake -G "MinGW Makefiles" ../src/brewtarget with no full path.

36 set(CPACK_OBJCOPY_EXECUTABLE "V:/msys64/ucrt64/bin/objcopy.exe") 37 set(CPACK_OBJDUMP_EXECUTABLE "objdump") 38 set(CPACK_OUTPUT_CONFIG_FILE "V:/msys64/usr/local/builducrt/CPackConfig.cmake")``

I also saw the note in CMakeLists.txt 1322 # I always hit a "Failed to start objdump" error.)

Nevertheless .exe size of direct install and package install are still way different.

[matty0ung]

Yes, I don't know what the heuristics are in CPack for locating these things. It's one of the pain points of using CPack: the automation saves you a bunch of effort when it works, but can be a bit cryptic to debug when it doesn't.

For the executable size, I'm also starting to wonder if the difference in size is to do with including/excluding debugging symbols.

[matty0ung]

This is from a Linux build, but shows the difference in size between a binary with and without debugging symbols:

$ ls -l packaging/linux/usr/bin/brewken 
-rwxr-xr-x 1 matt matt 128068752 Feb  5 15:50 packaging/linux/usr/bin/brewken
$ ls -l packaging/linux/brewken-0.1.0-1_amd64/usr/bin/brewken 
-rwxr-xr-x 1 matt matt 7003176 Feb  5 15:52 packaging/linux/brewken-0.1.0-1_amd64/usr/bin/brewken
$

[Melonbob]

Starting running my builds from last night with the current upstream this morning. My MINGW32 package build does not crash the App when exporting single hop and yeast xml records!! I'll test some more later.

Pretty clean log file and no Events logged. brewtarget.log

I had edited the full paths of build/CPackConfig.cmake from set(CPACK_OBJDUMP_EXECUTABLE "objdump") to set(CPACK_OBJDUMP_EXECUTABLE "V:/msys64/mingw32/bin/objdump.exe") after cmake finished configuring. I also found the full path missing from the same line in build/CPackSourceConfig.cmake but not until after it was already built.

What's your package build do now?

[matty0ung]

That sounds like good news! Am part-way through fiddling with the new build system, but when I've put everything back together again I'll give it another run.

[matty0ung]

Sorry for long radio silence on this. You should be able to build on Windows MSYS2 with meson, and then create packages using the bt build tool script (https://github.com/Brewtarget/brewtarget/blob/v3.0.8/bt). This is how we're doing nightly builds and recent releases. The bt script can also do almost all the package install etc you need prior to builds. There is still a bit of manual set-up (explained in comments in https://github.com/Brewtarget/brewtarget/blob/develop/.github/workflows/windows.yml), but it's not too much. First make sure you have Python installed, then:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

python get-pip.py

python -m pip install -U --force-reinstall pip

export PYTHONIOENCODING=utf8

if [[ ! -f $(dirname $(which python))/python3 ]]; then ln -s $(which python) $(dirname $(which python))/python3; fi          

./bt -v setup all

As ever, give us a shout with questions or problems, either by reopening this issue or raising a new one.