enhance Windows builds

[slayoo]

Thank's to Jeongbin we have now an initial set-up for continuous integration for windows! Let's discuss here potential enhancements:

• [ ] 32/64-bit build
• [x] test execution
• [ ] Python module
• [x] Debug/Release builds
• [ ] publishing artifact so that people could download a pre-build binary
• [ ] Cygwin build
• [x] Werror-like flag
• [ ] Werror flag for CMake

Work in progress can be monitored here: https://github.com/gnudatalanguage/gdl/pull/240

[maynardGK]

Regarding 64 bits, I had no trouble when I tried it using mingw/msys2. There was an error message that I got upon startup, though. When testing, it is easy to forget to turn off the customized procedures that will show up without resetting them. For suse/linux testing I do the following: export -n GDL_STARTUP export GDL_PATH=+/f/github/gdlgit Which is easy to forget on the windows side, since I use a separate CMD.exe/startup ritual.

For pre-built binaries, I worry about the legalities of distributing the support libraries without compliance with the GPL terms.

[slayoo]

Just for the record, please note that Appveyor offers the possibility to log in to the virtual machine during one hour after the build: https://www.appveyor.com/docs/how-to/rdp-to-build-worker/

[slayoo]

Perhaps we could simply package the pre-built packages together with the source code - in my understanding, this would make it legally OK, right?

[maynardGK]

@slayoo Appveyor scripts are based in PC language even though it looks a little like bash ("ls","cd","md" are built-ins, but that's about as far as it goes) whereas my environment in MSYS is basically "just" like linux. MSYS is installed on Appveyor, and there is an example the bash.exe invocation in the bsd-xdr build. Reminiscent of the punched-card days. I notice a linux mode that is probably the best route: make it with the mingw cross-compilers scripted in linux. The libraries at the Msys repositories are built with documented scripts that include origins, which is probably sufficient for the licensing terms. They include python, numpy, ... everything else but we do need specialized plplot, bsd-xdr builds. I haven't discovered yet how to unpack them in App, they all go to /mingw32 and /mingw64 trees. I'm unsure of where cmake runs out of, but it is best if it, too, resides in /mingw32 (/mingw64 for 64-bit builds).
http://repo.msys2.org/mingw/i686

My earlier comments are based on my brief experience with "make check"

[slayoo]

@maynardGK appveyor.yml instructions are not executed through PowerShell unless you prefix an entry with "ps: ". I'm all for having an MSYS build in addition to the current mingw one!

[pjb7687]

Hi all, I am trying to improve build script using MSYS. I once reported that the readline library included in MSYS is problematic in some machines, but now I found an alternative one for Windows which is called WinEditline, and now I guess everything can be built in MSYS. But at the moment I can't work on it and maybe I can do something in July.

[alaingdl]

@pjb7687 Graet ! We can help to test. Concerning WinEditLine I hope it will not have the same issue than (bsd) "editline" in OSX, which do NOT provide "history" ! (OSX editline functionalities are a subset of readline functionalities we are already using in GDL)

[Csega]

Could somebody show me a good tutorial about how to compile GDL on (native) Windows? I'd like to help improve Windows builds, however, so far I was unable to compile it on native Windows 10. Is there a way to do that as of now?

Also, I'd like to suggest vcpkg (https://github.com/Microsoft/vcpkg) to compile the dependencies (especially on Windows), which is a fully opensource, cross-platform "package manager", which uses CMake and compiles the packages from source. I added plplot (with a help of my friend), so in principle, every major dependency except readline/editline is available in vcpkg.

[maynardGK]

Hi Csega! I assume the microsoft packaging will be compiled with Visual Studio (MSVC) but we use mingw-w64 compilers and MSYS2 to provide the context, a posix shell. So it looks pretty much like what a Linux build does. Except for xdr and plplot, I think all needed support libraries can come from the repositories provided from msys2. General info for msys2: https://github.com/msys2/msys2/wiki In https://sourceforge.net/p/gnudatalanguage/discussion/338695/thread/4ebd1e60/ I go through most initial steps of setting up a new msys2 on a virgin (msw8.1) system, and building the support. I found I had to use readline 6.3 (instead of 7) and that GCC 7.2 was not compatible with eigen3.

@pjb7687 readline will work fine, stick with msys2 but see above re: eigen3, readline7

[slayoo]

For the record, we had also received reports of successful builds under cygwin and with mxe. Help with getting it to work in native environment would be greatly appreciated!

[Csega]

Hi @maynardGK!

Thanks for your answer! Vcpkg only uses MSVC under Windows, but as far as I know, even in Windows it can be used with LLVM (Clang) too. I try to look into it in detail. I also try to get it work under msys2 environment. I also tried to build GDL with msys2 a few months ago, but it did not work for me. I look into the details you sent me, hopefully I get it work this time.

@slayoo Do you have info about the mxe builds? I also looked into it, but the mxe system is too complex to me at a first glance. And as far as I know, a few packages are missing, however, not the most important ones.

[slayoo]

@GillesDuvert do you have any insights on the mxe builds?

[Csega]

Hi @maynardGK!

I tried to build GDL again with msys2, but now I remember, where I got stuck. I couldn't get GSL to work with GDL. I installed GSL with the following command:

pacman -S mingw32/mingw-w64-i686-gsl

But still, I got the following error message when building GDL (with cmake .):

  Gnu Scientific library (1.7 or higher) and libgslcblas are mandatory.

  Use -DGSLDIR=DIR to specify the gsl directory tree.

  (suitable Debian/Ubuntu package: libgsl0-dev)

  (suitable Fedora package: gsl-devel)

-- Configuring incomplete, errors occurred!

How should I install GSL in msys2? Should I open a separate issue for this?

[GillesDuvert]

regarding the mxe builds, I was able to create the chain on my linux doing as the site says. Then adding the following libraries that are needed and already 'mxe packaged' was easy. Long but easy. Note that it includes wxWidgets, eigen, glpk, gsl, imagemagick...

binutils_x86_64-w64-mingw32.shared gcc_x86_64-w64-mingw32.shared mxe-conf_x86_64-w64-mingw32.shared mingw-w64_x86_64-w64-mingw32.shared pkgconf_x86_64-w64-mingw32.shared ncurses_x86_64-w64-mingw32.shared libgnurx_x86_64-w64-mingw32.shared termcap_x86_64-w64-mingw32.shared readline_x86_64-w64-mingw32.shared zlib_x86_64-w64-mingw32.shared libpng_x86_64-w64-mingw32.shared gsl_x86_64-w64-mingw32.shared gmp_x86_64-w64-mingw32.shared glpk_x86_64-w64-mingw32.shared cmake-conf_x86_64-w64-mingw32.shared eigen_x86_64-w64-mingw32.shared jasper_x86_64-w64-mingw32.shared dbus_x86_64-w64-mingw32.shared freetype_x86_64-w64-mingw32.shared nettle_x86_64-w64-mingw32.shared gnutls_x86_64-w64-mingw32.shared bzip2_x86_64-w64-mingw32.shared ilmbase_x86_64-w64-mingw32.shared expat_x86_64-w64-mingw32.shared openexr_x86_64-w64-mingw32.shared libidn2_x86_64-w64-mingw32.shared vidstab_x86_64-w64-mingw32.shared opencore-amr_x86_64-w64-mingw32.shared theora_x86_64-w64-mingw32.shared opus_x86_64-w64-mingw32.shared speex_x86_64-w64-mingw32.shared imagemagick_x86_64-w64-mingw32.shared gettext_x86_64-w64-mingw32.shared ogg_x86_64-w64-mingw32.shared vorbis_x86_64-w64-mingw32.shared libunistring_x86_64-w64-mingw32.shared liblsmash_x86_64-w64-mingw32.shared libxml2_x86_64-w64-mingw32.shared fontconfig_x86_64-w64-mingw32.shared libcaca_x86_64-w64-mingw32.shared harfbuzz_x86_64-w64-mingw32.shared pthreads_x86_64-w64-mingw32.shared freeglut_x86_64-w64-mingw32.shared fftw_x86_64-w64-mingw32.shared libass_x86_64-w64-mingw32.shared jpeg_x86_64-w64-mingw32.shared libbluray_x86_64-w64-mingw32.shared libltdl_x86_64-w64-mingw32.shared x264_x86_64-w64-mingw32.shared libbs2b_x86_64-w64-mingw32.shared glib_x86_64-w64-mingw32.shared pcre_x86_64-w64-mingw32.shared sdl_x86_64-w64-mingw32.shared vo-amrwbenc_x86_64-w64-mingw32.shared lcms_x86_64-w64-mingw32.shared lzo_x86_64-w64-mingw32.shared lame_x86_64-w64-mingw32.shared icu4c_x86_64-w64-mingw32.shared libffi_x86_64-w64-mingw32.shared tiff_x86_64-w64-mingw32.shared libvpx_x86_64-w64-mingw32.shared yasm_x86_64-w64-mingw32.shared cairo_x86_64-w64-mingw32.shared pixman_x86_64-w64-mingw32.shared dlfcn-win32_x86_64-w64-mingw32.shared ffmpeg_x86_64-w64-mingw32.shared xvidcore_x86_64-w64-mingw32.shared fribidi_x86_64-w64-mingw32.shared libiconv_x86_64-w64-mingw32.shared freetype-bootstrap_x86_64-w64-mingw32.shared xz_x86_64-w64-mingw32.shared portablexdr_x86_64-w64-mingw32.shared hdf4_x86_64-w64-mingw32.shared hdf5_x86_64-w64-mingw32.shared curl_x86_64-w64-mingw32.shared netcdf_x86_64-w64-mingw32.shared libgpg_error_x86_64-w64-mingw32.shared libgcrypt_x86_64-w64-mingw32.shared libssh2_x86_64-w64-mingw32.shared proj_x86_64-w64-mingw32.shared wxwidgets_x86_64-w64-mingw32.shared pango_x86_64-w64-mingw32.shared

... but not plplot, that can be compiled from the sources using:

x86_64-w64-mingw32.shared-cmake -Wno-dev -DCMAKE_NATIVE_BINARY_DIR="$HOME/PACKAGES/plplot-5.11.1/linux" -DCMAKE_CXX_FLAGS="-fno-keep-inline-dllexport -std=gnu++11" -DDEFAULT_NO_DEVICES:BOOL=ON -DENABLE_cxx=ON -DDEFAULT_NO_BINDINGS=ON -x86_64-w64-mingw32.shared-cmake -Wno-dev -DCMAKE_NATIVE_BINARY_DIR="$HOME/PACKAGES/plplot-5.11.1/linux" -DCMAKE_CXX_FLAGS="-fno-keep-inline-dllexport -std=gnu++11" -DDEFAULT_NO_DEVICES:BOOL=ON -DENABLE_cxx=ON -DDEFAULT_NO_BINDINGS=ON -DPLD_ps=ON -DPLD_svg=ON -DPLD_mem=ON -DPLD_null=ON -DENABLE_wxwidgets=ON -DPLD_wxwidgets=ON -DENABLE_DYNDRIVERS=OFF -DBUILD_TEST=ON ..
DPLD_ps=ON -DPLD_svg=ON -DPLD_mem=ON -DPLD_null=ON -DENABLE_wxwidgets=ON -DPLD_wxwidgets=ON -DENABLE_DYNDRIVERS=OFF -DBUILD_TEST=ON ..

plplot is probematic as the (cross)cmake complains but in fact "make" afterwards works sufficiently to get the library. We do not need any recent plplot, also, but 5.11 was ok . (my reference is 5.9.9, why, we use only 1% of plplot). Then GDL needs a bit of tweaking. The first bold attempt with: x86_64-w64-mingw32.static-cmake -Wno-dev -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS_DEBUG='-g -Wa,-mbig-obj' -DPYTHON=OFF -DPSLIB=OFF -DGLPK=ON -DLIBPROJ4=ON -DNETCDF=OFF -DWXWIDGETS=OFF ../gdl-test ...fails. I got it compiled with: 86_64-w64-mingw32.shared-cmake -Wno-dev -DCMAKE_CXX_FLAGS="-DUNICODE -D_UNICODE -Wno-deprecated-declarations -fno-keep-inline-dllexport " -DPYTHON=OFF -DPSLIB=OFF -DGLPK=OFF -DLIBPROJ4=ON -DNETCDF=OFF -DWXWIDGETS=ON ../gdl_test With proper changes in gdl/CMakeLists.txt CMakeLists.txt and gdl/src/CMakeLists.txt CMakeLists.txt

Then the linking is not straightforward as the -l libraries are not there, or not in the good order. Play around this list (not guaranteed): antlr/libantlr.a -lws2_32 -Wl,-Bstatic -lform -lreadline -lhistory -lz -lpng -lz -Wl,-Bdynamic -fopenmp -Wl,-Bstatic -lgsl -lgslcblas -lplplotcxxd -lplplotd -lgdi32 -lcomdlg32 -lcsirocsa -lqsastime -lMagick++-6.Q16 -lMagickWand-6.Q16 -lMagickCore-6.Q16 -llcms2 -ljpeg -llqr-1 -lfftw3 -lharfbuzz -lcairo -lgobject-2.0 -lfontconfig -lusp10 -lmsimg32 -lpixman-1 -L$HOME/PACKAGES/mxe/usr/x86_64-w64-mingw32.static/lib/../lib -lffi -lexpat -lfreetype -lpng16 -lharfbuzz_too -lfreetype_too -lglib-2.0 -lws2_32 -lole32 -lwinmm -lshlwapi -lpcre -lintl -liconv -llzma -lbz2 -lIlmImf -lImath -lHalf -lIex -lIexMath -lIlmThread -lz -lgdi32 -lglpk -lproj -lhdf5_hl -lhdf5 -lmfhdf -ldf -lz -lfftw3l -lfftw3f -lncurses -lgnurx -lpsapi -lxdr -lws2_32 -ltermcap -lm -lkernel32 -luser32 -lwinspool -lshell32 -loleaut32 -luuid -ladvapi32 Normally, doing "wine gdl.exe" should then have done something. Instead it was stuck. Idem on a windows 7 machine. I stopped then and went back to linux affairs...

[maynardGK]

@Csega You probably have some windows environment mixed in that you don't want. Edit the /etc/profile to clean out the windows' portion of the PATH to the bare minimum; add it to the end of the appropriate MINGW32/MSYSTEM or MINGW64 path.

Run CMake from the installed /mingw32/bin/cmake - for any version later later than 3.1 or so then everything you installed should be found. Also don't bother trying to use windows' python.

[slayoo]

I will add the mxe build to Travis, there are actually ready-to-use examples, e.g.: https://github.com/starius/battleship/tree/master/build

[GillesDuvert]

good luck, as this is not a simple battleship program...

[Csega]

Thanks guys for the help! Unfortunately I can only try this on Monday, but I'll report the results!

@alaingdl I tried WinEditLine and there is a history. :)

[Csega]

Hi Guys!

@GillesDuvert I tried to build MXE on MSYS2, but I cannot find (I tried already search for it on Google, but still nothing) the gdk-pixbuf-csource depencency. Does anyone know which package contains it? gdk-pixbuf2 and gtk2 are not the right answer. I also tried to just install MXE on MSYS2, but I could not find it as an installable package.

@maynardGK I checked /etc/profiles, but I couldn't figure out, what to delete. It's the factory routine, it seems intact. I tried to use /mingw32/bin/cmake, but it did not find readline (or editline), however, both packages are installed both for 32 and 64 bit...

[maynardGK]

@Csega (imo) mxe is probably a linux thing, or for cygnus, to compile and assemble the code with mingw compilers. I'm only suspecting that PATH is your problem, your windows' system may have put in there (by installed programs) many names that could cause conflicts, so what I have in my /etc/profile at the top # To learn more about startup files, refer to your shell's man page. OLDPATH=${PATH} export -n PATH_SEPARATOR export PLPLOT_DRV_DIR=C:/usr/plplot/drivers export PLPLOT_HOME=C:/usr/plplot export GITHOME=/f/github export GDLGIT=/f/github/gdlgit C=/usr/bin/cygpath $SYSTEMDRIVE wpath="$C/Windows:$C/Windows/system32:$C/Windows/system32/Wbem" MSYS2_PATH="/usr/local/bin:/usr/bin:/bin"

the PLPLOT symbols point to directories in support of running PLPLOT: try running without them, if you succeed tell me about it! (the plplot group would have you believe they are not needed).

[GillesDuvert]

@Csega , indeed mxe is a "toolchain" to build windows .exe file on linux only.

[pjb7687]

Hi all, I would like to start working on the MSYS build as mentioned before, but I have no idea from which branch should I start work on. Should it be master branch? Or based on this pull request? https://github.com/gnudatalanguage/gdl/pull/408

[maynardGK]

@pjb7687 Hi JP, If you can pull the #408 then scripts/appveyor_1.msys and scripts/appveyor_plplot.msys are good starter guides to the /mingw32 you need. Also the simple patch to FindPCRE.cmake: find_library(PCRE_LIBRARY NAMES systre pcre) will bring in the systre system (unavailable to linux, and so compatible).

[pjb7687]

@maynardGK Thanks. May I ask why systre is more preferable than pcre?

[alaingdl]

Here, we are proud to say Sylvain & Amar just succeed to have a GDL working on MSwin (some libs. are missing, but plotting OK :)

More details to come ASAP !

[maynardGK]

@pjb7687 I think pcre won't have the functions requested (for readline > 5)

The current status of #408 is, it is building plplot-5.13.0 using MSYS which is hooked up with GCC-7.3 -(I don;t know how to switch in a different GCC to match what else is done). This plplot has a wxwidgets driver (previous appveyor builds using mingw32-make did not succeed). In the current version, mingw32-make is then used to build gdl (without wxWidgets; doing so with widgets results in a gdl.exe that will crash when demo_widgets is attempted.) In Appveyor, the build crashes 95% of tests when "mingw32-make check" is run (build 524). So building it and not running "mingw32-make check" is done to make a viable GDL: download gdl_build.zip here: https://ci.appveyor.com/project/slayoo/gdl/build/1.0.525

This runs the testsuite tests that it failed in mingw32-make during the previous check.

In the msys script that makes plplot, an attempt is made to create a WIN32-compatible wxwidgets support from the msys repository distribution. The attempt does not work, thus far.

In #408, I next plan to implement the msys build of gdl. The script is ready, for wxwidgets it will also involve use of the FindwxWidgets.cmake version that is used in the plplot build, that for the "unix style" build a passed path "-DWXWIDGETS_USE_PREFIX=" that will then be used in all subsequent usage of the "bin/wx-config" command. In addition, the wx-config will require a patch to modify the link parameters. Under normal circumstances, "wx-config" gives the correct answers because it has been edited to reflect its location; they come wired as C:/msys64/mingw32 but we cannot write into that directory.

[slayoo]

Closing this one, as a lot has been accomplished by @maynardGK with Appveyor and perhaps creating new issues for particular issues (like #615) will be more informative.