Cross compile error when using mingw.

[hwiorn]

https://github.com/kkm000/openfst/blob/338225416178ac36b8002d70387f5556e44c8d05/CMakeLists.txt#L19-L28

GNU compiler doesn't process command switch starting with /. So mingw can't process the /bigobj option. I think that WIN32 need to be changed to MSVC or wrap that add_definitions block with if(MINGW) ... endif(). The condition seems to be only for MSVC compiler.

[  0%] Building CXX object src/lib/CMakeFiles/fst.dir/compat.cc.obj
x86_64-w64-mingw32-g++-posix: error: /bigobj: No such file or directory
make[2]: *** [src/lib/CMakeFiles/fst.dir/build.make:77: src/lib/CMakeFiles/fst.dir/compat.cc.obj] Error 1
make[1]: *** [CMakeFiles/Makefile2:1280: src/lib/CMakeFiles/fst.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

When I changed to WIN32 to MSVC simply at line 19, it compiled ok.

[kkm000]

@hwiorn: Thanks for the recommendation. That's probably the correct way to do it. Let me take care of this.

A question, out of pure curiosity: I never tried MinGW. I am wondering what benefits does it provide? Microsoft cl is one of the best, if not the best optimizing compiler out there, and is available free of charge¹. I understand MinGW is a port of GNU g++ and its glibc to Windows, is this approximately correct? Are you using because you need a library which is compatible with something else compiled with the same compiler, or does it produce object/library files compatible with MSVC?

---

¹ "Valid license" refers to any SKU EULA, including the free Community Edition. Last time I installed them on a CI server the package did not ask for anything except agreeing with the license. Hope that haven't changed.

[hwiorn]

I'm sure and agree your opinion that VC(IDE, tools) and cl are the best option on Windows system. I've developed some programs on Windows and I know there is a VS community edition(including VC++ express version) already. I used VC express(now community) version on Windows and I saw VC can optimize better than GCC in some area. So I always recommend it on Windows to my colleagues.

On the other hand, MS provides free version of compiler but Windows is not. And they don't provide a VC(especially cl) for Linux and It means I have to manage a Windows machine(physically or virtually) to compile and maintain the system. MS provides Windows development virtual machine for evaluation and testing. I'm not sure that I can use it for business legally. Last time I saw, I couldn't use it on business.

In my experiences of supporting cross-platform, It's really annoyed to write for different compiler on different systems which I have to compile on both system every time when I changed code. I was trying to make Jenkins to build it for Windows; It didn't work sometimes where I have to maintain the system with Jenkins settings. Eventually I changed from Jenkins to Virtual Box and QEMU/KVM. I know there is a fancy option called WSL. But it is like Linux that needs to cross compile to make windows executables like Cygwin and MSys2(+MinGW) in my experience.

I make programs on Linux using Python, Java, C++ and Go. That languages can cross-compile for Windows using MingGW on Linux. Go have options to build executables for different OSes and architectures with CGO. It means I can handle CI/CD pipeline using Linux containers, just without bring it into other platforms. My purpose is make same(or similar) results using MinGW without MS cl. I guess the libraries compatible with MSVC is next thing which my colleagues may request me.

Recently, I cross-compiled Kaldi for Windows using cmake and mingw successfully to use in a program built by Go . I could have fixed openfst source code just for my purpose privately. But I think the open-source have a new option is always good.

This is just my experience, You can correct me if I'm wrong :)

[hwiorn]

The patch can be like the below. It needs -Wa,-mbig-obj with optimization level like -O2. I tested many times. I don't know why it makes error when no optimization flags.

diff --git a/CMakeLists.txt b/CMakeLists.txt
index f6189db..dcbd2de 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,8 +17,12 @@ set(CMAKE_MACOSX_RPATH 1)
 set(CMAKE_CXX_STANDARD 11)

 if (WIN32)
-  add_definitions(/bigobj)
-  set(WHOLEFST "/WHOLEARCHIVE:fst")
+  if (MSVC)
+    add_definitions(/bigobj)
+    set(WHOLEFST "/WHOLEARCHIVE:fst")
+  else()
+    add_compile_options(-Wa,-mbig-obj -O2)
+  endif ()
   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${WHOLEFST}")
   #set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS 1)
   #this must be disabled unless the previous option (CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS) is enabled

[kkm000]

@hwiorn, thanks for the patch! The -mbig-obj option is required due to differences in the way compiler generates sections in object files. The historical limit on the number of sections in the PE/COFF format is around 2^16, and heavily templated libraries may exceed that limit. /BIGOBJ (msvc) or -mbig-obj (MinGW) enables up to 2^32 sections. It's just a compatibility setting, only one 16-bit word replaced with a 32-bit one in the header, but that is incompatible with very old linkers. I think (but am not sure) this is now the default for C++ applications; it certainly is for UWP apps.

I was not implying that you do something wrong, and I'm sorry if you read my question this way. I was just curious why do people use MinGW, and estimate if its use is common enough for us to support it. I didn't ever realize you could cross-compile programs for Window with it on Linux! I understand you, build tools are free (and you do not even need the Community edition, only Build Tools), but the OS itself isn't!

When I bought mine in 2015 or 16 it was cheap ($200 for a license transferable between computers as many times as you want, so I picked up a Pro edition "just in case"; because Windows 10 is fully supported till 2025, it came to $20/yr TCO which is basically nothing. But now 2025 and EOL is much closer :) ). AFAIK, Windows 10 does not expire or die if you run it without a license, only keeps very mildly nagging you, but generally you need a license. Microsoft had to make a lot of concessions after the Ballmer era has ended—the dude nearly destroyed the internal culture and smeared company public image. Nadella had to do major repairs: the house was falling apart, and he have done a very good job. Thus. I don't expect such a generosity with Windows 11, but we'll see! :)

Thanks again for the feedback and the patch!

[hwiorn]

heavily templated libraries may exceed that limit

I couldn't think about template optimizations! Thank you! I forgot that openfst is heavily template library. Other template library like fmt uses -mbig-obj option when it uses mingw.

I was just curious why do people use MinGW, and estimate if its use is common enough for us to support it. I didn't ever realize you could cross-compile programs for Window with it on Linux!

I think cross compiling is used when target system have limited (hardware and software) resource or some reasons which can be philosophy or avoiding license issue.

Even not using dependent code and libraries like MFC, ATL and COM, writing code on both system made me feel like I am in never ending hell sometimes when I met compile error with large code changed(10+ commits) on another system. It means I had to fix errors and logics while switching both system where make mistakes easily. Some compiler makes different template result and some compiler is scolding it.

I use self-hosted gitlab and gitlab CI/CD which can run it on docker containers. Container system like docker and k8s is based on Linux stack and it makes possible to build projects and release the files without switching another system.

Windows 10 does not expire or die if you run it without a license, only keeps very mildly nagging you, but generally you need a license.

I know how I can stop Windows is nagging me because I was a Windows fanboy ;) After I accustomed macOS and Linux, I've used them(especially Linux now) more than Windows. Actually, I have Windows 10 Pro licenses personally, so the license can not make a problem for me. But I experienced some small business company hasn't been affordable for Windows licenses and physical machine powers enough on their business.

Thanks again for the feedback and the patch!

You're welcome. Thank you for your kind response.

[Gearspre]

@hwiorn hello, I'am tring to use i686-w64-mingw32 to compile this project. And use your patch, but still causing error. error message: ` src/include/fst/float-weight.h:99:2: error: #error "Please compile with -msse -mfpmath=sse, or equivalent."

error "Please compile with -msse -mfpmath=sse, or equivalent."

^~~~~ src\lib\CMakeFiles\fst.dir\build.make:105: recipe for target 'src/lib/CMakeFiles/fst.dir/fst-types.cc.obj' failed make[2]: [src/lib/CMakeFiles/fst.dir/fst-types.cc.obj] Error 1 CMakeFiles\Makefile2:1278: recipe for target 'src/lib/CMakeFiles/fst.dir/all' failed make[1]: [src/lib/CMakeFiles/fst.dir/all] Error 2 Makefile:144: recipe for target 'all' failed make: *** [all] Error 2 ` Can you show the CMakeList.txt after fixed? Thanks for your reply!

[hwiorn]

`src/include/fst/float-weight.h:99:2: error: #error "Please compile with -msse -mfpmath=sse, or equivalent." #error "Please compile with -msse -mfpmath=sse, or equivalent.

Actually It's not related with the MinGW. You are using i686-w64-mingw32 (for 32bit target). Did you compile with -msse2 -mfpmath=sse?

You can see the conditional-macro inside float-weight.h and It checks __SSE2_MATH__ not __SSE_MATH__.

https://github.com/kkm000/openfst/blob/338225416178ac36b8002d70387f5556e44c8d05/src/include/fst/float-weight.h#L94-L100

You should specify that compile flags when you compile for the 32-bit target. You can see the exactly same checking in the configure file. X86-64 GCC uses SSE and SSE2 default. You can see this on the GCC x86-options document.

For the x86-32 compiler, you must use -march=cpu-type, -msse or -msse2 switches to enable SSE extensions and make this option effective. For the x86-64 compiler, these extensions are enabled by default.

You can check your macro conditions like below.

# Mingw 64-bit GCC-10
$ /usr/bin/x86_64-w64-mingw32-g++-posix -dM -E - </dev/null | grep SSE
#define __MMX_WITH_SSE__ 1
#define __SSE2_MATH__ 1
#define __SSE_MATH__ 1
#define __SSE__ 1
#define __SSE2__ 1

# Mingw 32-bit GCC-10
$ /usr/bin/i686-w64-mingw32-g++-posix -dM -E - </dev/null | grep SSE

# MinGW 32-bit GCC-10 with -msse2 -mfpmath=sse
$ /usr/bin/i686-w64-mingw32-g++-posix -msse2 -mfpmath=sse -dM -E - </dev/null | grep SSE
#define __SSE2_MATH__ 1
#define __SSE_MATH__ 1
#define __SSE__ 1
#define __SSE2__ 1

Can you show the CMakeList.txt after fixed?

CMakeLists.txt is not matter in your case. You need to set the compile flags for 32-bit. :)

[kkm000]

I experienced some small business company hasn't been affordable for Windows licenses and physical machine powers enough

You need much more of a hardware oomph from the build machine than the absolute minimums for either OS. Take that into account, and the difference disappears. And a business that can't afford an extra few hundred bucks is not anywhere close to being viable anyway.

Tangentially,

I don't expect such a generosity with Windows 11, but we'll see! :)

Yay, I was wrong again!

Srsly, @jtrmal, do you think we want to support cross-compile on Linux for Windows, or not? I can't grasp the idea. You cross-compile binaries for Windows on a free-as-beer-isn't OS, and then... what? Release without testing? God, no. Use them without Windows? But how? I'm lost. Maybe we should, but I currently don't see any benefits. Someone else may fork and do it.

[jtrmal]

I'm more inclined setting up a github actions to make openfst releases compiled with different compilers cross-compiling is pain y.

On Sat, Sep 3, 2022 at 2:24 AM kkm000 @.***> wrote:

I experienced some small business company hasn't been affordable for Windows licenses and physical machine powers enough

You need much more of a hardware oomph from the build machine than the absolute minimums for either OS. Take that into account, and the difference disappears. And a business that can't afford an extra few hundred bucks is not anywhere close to being viable anyway.

Tangentially,

I don't expect such a generosity with Windows 11, but we'll see! :)

Yay, I was wrong again!

Srsly, @jtrmal https://github.com/jtrmal, do you think we want to support cross-compile on Linux for Windows, or not? I can't grasp the idea. You cross-compile binaries for Windows on a free-as-beer-isn't OS, and then... what? Release without testing? God, no. Use them without Windows? But how? I'm lost. Maybe we should, but I currently don't see the benefits. Someone else may fork and do it.

— Reply to this email directly, view it on GitHub https://github.com/kkm000/openfst/issues/42#issuecomment-1235999014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYX4Y4RFTWCONPWP2XYDV4KLD3ANCNFSM5DUXGBAQ . You are receiving this because you were mentioned.Message ID: @.***>

[hwiorn]

@kkm000

You need much more of a hardware oomph from the build machine than the absolute minimums for either OS. Take that into account, and the difference disappears. And a business that can't afford an extra few hundred bucks is not anywhere close to being viable anyway.

No, For cross compiling use-case and licenses you asked, It were the small companies's issue I saw. In this case, It's not my money issue. I have already valid Windows 10 licenses. I'm not saying poorman's cross-compiling. For me, Developing on Linux is my preference now and I accustomed Linux tools than MSVC now. You know that, using Linux(Unix) tools on Windows is really slower and somtimes pain. So I tried to cross compile for Windows on Linux. Because it works. And It can make CI/CD pipeline docker based easily than Windows VM basded. As I said, I don't like to fix and test core logic many things for different targets on one source code base after reboot to target platform. Because It makes side effects easily I think.

Srsly, @jtrmal, do you think we want to support cross-compile on Linux for Windows, or not? I can't grasp the idea. You cross-compile binaries for Windows on a free-as-beer-isn't OS, and then... what? Release without testing? God, no. Use them without Windows? But how? I'm lost. Maybe we should, but I currently don't see any benefits. Someone else may fork and do it.

On Linux, You can test with wine and wine64 before release on Windows no reboot and VM. Sometimes, wine can't be enough if you use specific WinAPIs or host DLLs. I tested, kaldi decoder was enough this case. So When I modify core logic, I can test and see it will work on two different OS right.

@jtrmal

I'm more inclined setting up a github actions to make openfst releases compiled with different compilers cross-compiling is pain

I'm agreed with you. cross-compiling is REALLY pain.

I think OpenFST doesn't have platform specific features (such as CUDA, GUI, WDK, DXD, Services and etc.). OpenFST side for MinGW needs this CMake patch only I pasted.

diff --git a/CMakeLists.txt b/CMakeLists.txt
index f6189db..dcbd2de 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,8 +17,12 @@ set(CMAKE_MACOSX_RPATH 1)
 set(CMAKE_CXX_STANDARD 11)

 if (WIN32)
-  add_definitions(/bigobj)
-  set(WHOLEFST "/WHOLEARCHIVE:fst")
+  if (MSVC)
+    add_definitions(/bigobj)
+    set(WHOLEFST "/WHOLEARCHIVE:fst")
+  else()
+    add_compile_options(-Wa,-mbig-obj -O2)
+  endif ()
   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${WHOLEFST}")
   #set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS 1)
   #this must be disabled unless the previous option (CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS) is enabled

I use CMake fetchcontent_declare and PATCH_COMMAND for MinGW Kaldi patching my own.

fetchcontent_declare(
    kaldi
    GIT_REPOSITORY "https://github.com/kaldi-asr/kaldi" 
    GIT_TAG ac29a6ff09823d1cbb4814da60360c966f33cd0d
    UPDATE_DISCONNECTED 1
    PATCH_COMMAND git apply ${CMAKE_CURRENT_SOURCE_DIR}/patches/kaldi-mingw.patch
)
fetchcontent_makeavailable(kaldi)

I had decided to make MinGW patch for Kaldi because Kaldi supports Cygwin compile and I thought It was possible with a little modification.

OpenFST is impossible to be patch by this way. its src is generated by openfst.cmake of Kaldi. So I opened this issue.

https://github.com/kaldi-asr/kaldi/blob/7bc53ef85f1076b167c9e15216dc97081899cd07/cmake/third_party/openfst.cmake#L4-L8

I'm going to close this issue. If Someone want trying to cross-compile OpenFST for Windows, you can apply the patch I pasted.

[kkm000]

@hwiorn, this may be a good idea, thanks. We can put the patch into contrib/, refer to this discussion and note it's unmaintained.

I'm gonna rid of the /wholearchive thing. Interesting, the upstream does not compile as an archive (.a), only as an SO, for the same reason this had to be added (an archive member is not even linked in a static build b/c nothing refers to its external symbols, but must be linked into an executable binary, as it registers basic FST kinds in its static initialization, such as vector o_O. CMake has what they call a "unity mode" when it concats a specified group of sources into a temporary single source and compiles it into a single .o/.obj file. I did that for another project already. SO/DLL does not have this problem, as they do the initialization of static BSS variables, and that object is file always linked in.

Often, CMake supports features without the last-resort constructs like "if this compiler, set this, else set this". I'll check if the bigobj thing is one of the properties. It's an MS-only thing, AFAICR, number of entries in a table of some file type (obj? lib?) was limited to 65536 or a few below, so they extended the format; still, it may be exposed by CMake. -O2 is added by selecting a Release build; hard-coding it is superfluous at best, and makes code practically undebuggable.

@jtrmal I have a feeling we'll do good to everyone if we add CMake tooling to our vanilla downstream, too. I'm a fan. It's ugly, naturally, but I think I tried all xplat build systems. Bazel firmly holds the first place as the most impenetrable build system of all. BTW, CMake's support for VS is especially horrible. I've got an impression that they generate build files from templates with very little understanding what they really do. That arguably had been sensible for VS 2010 and earlier (.vcproj); later versions integrated with MSBuild without quirks (.vcxproj). Probably build is just hard.

I didn't grok the idea about binary releases. N(compilers)×M(arch,bitness,CPU)×2(static,dynamic)×2(debug,release) is a large number of builds. [*not all platforms]. N(compilers) may be reduced to N'(ABIs), but that's still a buttload of builds.

BTW, on the same note, drop or keep VS 2017 support?

[jtrmal]

yeah, pretty much the whole built matrix. should be easy to do using the github actions. Would probably prune on arch and cpu and compilers probably only VS 2019 and 2022 y.

On Mon, Sep 5, 2022 at 12:45 PM kkm000 @.***> wrote:

@hwiorn https://github.com/hwiorn, this may be a good idea, thanks. We can put the patch into contrib/, refer to this discussion and note it's unmaintained.

I'm gonna rid of the /wholearchive thing. Interesting, the upstream does not compile as an archive (.a), only as an SO, for the same reason this had to be added (an archive member is not even linked in a static build b/c nothing refers to its external symbols, but must be linked into an executable binary, as it registers basic FST kinds in its static initialization, such as vector o_O. CMake has what they call a "unity mode" when it concats a specified group of sources into a temporary single source and compiles it into a single .o/.obj file. I did that for another project already. SO/DLL does not have this problem, as they do the initialization of static BSS variables, and that object is file always linked in.

Often, CMake supports features without the last-resort constructs like "if this compiler, set this, else set this". I'll check if the bigobj thing is one of the properties. It's an MS-only thing, AFAICR, number of entries in a table of some file type (obj? lib?) was limited to 65536 or a few below, so they extended the format; still, it may be exposed by CMake. -O2 is added by selecting a Release build; hard-coding it is superfluous at best, and makes code practically undebuggable.

@jtrmal https://github.com/jtrmal I have a feeling we'll do good to everyone if we add CMake tooling to our vanilla downstream, too. I'm a fan. It's ugly, naturally, but I think I tried all xplat build systems. Bazel firmly holds the first place as the most impenetrable build system of all. BTW, CMake's support for VS is especially horrible. I've got an impression that they generate build files from templates with very little understanding what they really do. That arguably had been sensible for VS 2010 and earlier (.vcproj); later versions integrated with MSBuild without quirks (.vcxproj). Probably build is just hard.

I didn't grok the idea about binary releases. N(compilers)×M(arch,bitness,CPU)×2(static,dynamic)×2(debug,release) is a large number of builds. [*not all platforms]. N(compilers) may be reduced to N'(ABIs), but that's still a buttload of builds.

BTW, on the same note, drop or keep VS 2017 support?

— Reply to this email directly, view it on GitHub https://github.com/kkm000/openfst/issues/42#issuecomment-1236837121, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYXYCB7FTYX7Y3GEFHA3V4XFMLANCNFSM5DUXGBAQ . You are receiving this because you were mentioned.Message ID: @.***>

[kkm000]

@jtrmal, some algos are pretty intensive. Interesting how SSE vs AVX vs AVX2 affects the perf. May be irrelevant, may be not. They aren't too obsessed with perf.

I had to apply a simple patch to shave 30%(!) from the HCLG build time. Since the graph had to be done preferably while the TTS was still flapping its jaw (if the caller spoke, I buffered, but shut-up-on-VAD was off until the recognizer was ready). You'd never guess where all the time had gone! :) Encode() and Decode() use the default initial capacity of the std::unordered_map, the one that stores weights—as many as there are edges. Starting with the default capacity (16, I think), and doubling it with a realloc as needed did not quite made it too efficient with millions of edges :) It's amazing how much seriously algorithmically complex stuff is going on, and you remember the docs, every op has a specified complexity, including even undecidable. And these two O(1) operations stole the 1/3 of the total runtime!

[jtrmal]

true, but I see the releases mostly to help people get something done. If they will need super performance, they should compile and optimize specifically for their setup. Let me know wdyt y.

On Mon, Sep 5, 2022 at 1:35 PM kkm000 @.***> wrote:

@jtrmal https://github.com/jtrmal, some algos are pretty intensive. Interesting how SSE vs AVX vs AVX2 affects the perf. May be irrelevant, may be not. They aren't too obsessed with perf.

I had to apply a simple patch to shave 30%(!) from the HCLG build time. Since the graph had to be done preferably while the TTS was still flapping its jaw (if the caller spoke, I buffered, but shut-up-on-VAD was off until the recognizer was ready). You'd never guess where all the time had gone! :) Encode() and Decode() use the default initial capacity of the std::unordered_map, the one that stores weights—as many as there are edges. Starting with the default capacity (16, I think), and doubling it with a realloc as needed did not quite made it too efficient with millions of edges :) It's amazing how much seriously algorithmically complex stuff is going on, and you remember the docs, every op has a specified complexity, including even undecidable. And these two O(1) operations stole the 1/3 of the total runtime!

— Reply to this email directly, view it on GitHub https://github.com/kkm000/openfst/issues/42#issuecomment-1236883200, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYX6LLHB7EYF33DUTHULV4XLJVANCNFSM5DUXGBAQ . You are receiving this because you were mentioned.Message ID: @.***>

[kkm000]

@jtrmal with this idea in mind, certainly.

Wait, I used to release the whole kaboodle, now doing only .exe files. But VC compiler ABI is much more stable now, so we can do the libs too.