Build system

[danpovey]

For the new version of Kaldi, does anyone think we should switch to a different build system, such as cmake? We should probably still have manually-run scripts that check the dependencies; I am just wondering whether the stuff we are doing in the 'configure' script would be better done with cmake, and if so, whether anyone is interested in making a prototype to at least let it compile on Linux.

[kkm000]

I am not a big fan of CMake. @jtrmal is should have a more informed opinion, as he spend some time playing with it. To me, it looks more cryptic than gmake.

If we are open to any options at this point, we may consider Bazel. It is certainly different from other build systems, but it is very good at what it does.

Also, guys, let's please re-consider transitioning to Google Test seriously. E. g., when rewriting the logging code, I had to eyeball log output vs having progam expected output tests do it for me, or missed their so called "death test" (like you throw an unhandled exception, and the test succeeds when the SoT dies--it forks a copy of itself that is heading to seppuku internally, so zero set up for that, too). And it even comes in a single-header-only variant, and is so mature that it's updated once in a few years, and super-portable, even to Windows.

[kkm000]

Let me clarify on CMake. On Windows, it is doing a super-lame thing. It generates project and solution files from some templates, as if someone did not really put any effort into understanding what is in these files. It just stuffs filenames into templates that were apparently treated as blackbox. I do not want to disparage anyone and call them out for not applying any effort into making CMake a decent system on all platforms; the task is indeed daunting. But, in the end, it supports over 9000 platforms... more or less. When you have one project, it's OK. When you have so many files and libraries as Kaldi does, and want to change just one little aspect of it (e. g. suppress one warning), you are either up to editing 50 project files or regenerating everything from the grounds up. In make, I just add CXXFLAGS=, and Bob's yer uncle. It's not what I would rate as best feature of a build system. This it's inherent property of being a "meta-build" system, not a build system sensu stricto, that I dislike.

[sikoried]

I find cmake better than plain make, mostly for the support how to include dependencies. However, it's also a bit of a pain to wrap your head around it. I've heard good things about bazel, in particular regarding build servers and incremental build (both on server and client).

[jtrmal]

What I like about cmake because it's fairly expressive (in the sense that even if you are not familiar with it, you are able to figure out what most of the commands do). It has a good dependency tracking system (probably make depends wouldn't be needed anymore). It fairly widely accepted (people know it) and is still being developed and supported. I agree it does weird things when generating build files for Visual Studio. But it seems MS has the intention to support CMake in the VS. At least I've heard something for VS2017, not sure if it happened or if they dropped it or what -- @kkm would probably know. Big project do use cmake -- I mean not all of them, but at least some of them -- I know about KDE and LLVM/Clang. Clang was very painless to build for me (compared to gnu compilers). That might not be because of Cmake/Make though I think Kaldi would be very easy to convert to CMake -- as a matter of fact, I wrote a perl script, which converts the Kaldi makefiles into CMakeLists.txt. Also, the OpenFST was very easy to convert to CMake.

Other options are Bazel (as Korbinian said), scons (uses python) or jam (which I know was/is used in boost) I have no experience with Bazel. I've found jam (or bjam?) quite annoying when trying to build boost. I didn't spend too much time on scons y.

On Mon, Mar 11, 2019 at 6:35 AM Korbinian notifications@github.com wrote:

I find cmake better than plain make, mostly for the support how to include dependencies. However, it's also a bit of a pain to wrap your head around it. I've heard good things about bazel, in particular regarding build servers and incremental build (both on server and client).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471487274, or mute the thread https://github.com/notifications/unsubscribe-auth/AKisX1PnEkwWfTxp5-Z_yI0TboPDCxvwks5vVjF6gaJpZM4bnduF .

[kkm000]

it seems MS has the intention to support CMake in the VS. At least I've heard something for VS2017, not sure if it happened or if they dropped it or what -- @kkm would probably know.

They do. Which does not mean you can open a Kaldi-sized project with it. Since CMake generates about 500 separate project files, one per library and one per exe, it just blows up and dies spectacularly. For smaller projects, like zlib, it worked okay... It's not even MS to blame this time, it's how CMake handles Visual Studio. One output, one separate project in a separate directory.

I do not care much about building the whole rig on Windows. I tried it once, and it was less that exciting in the end. I even ran an eg end to end (tedlium, IIRC), using Cygwin bash. I even added support for it to Kaldi code, so things like popen pipe stuff through bash.

Windows 10 has added two intereting features. One is real functioning symlinks (in fact, they were there in NTFS all the time, but with a twist: you needed to be an admin to create symlinks, and, while you can grant any account a separate privilege to create symlinks, this privilege is unconditionally suppressed for admin accounts, unless you elevate, i. e. "sudo". So, to translate to Linux lingo, you must either be either a non-sudoer or create symlinks under sudo. In W10, all you need is enable developer mode).

The second is native support for Linux user-mode binaries. This is an interesting way to go, but I am not sure if this will cut it for Kaldi. I'll try at some point. It's an interesting Linux. It actually runs in a kind of container, with a special init. Everything in else in the usermode is just a normal Linux, with ELF binaries running out of the box. There are a few distros available, including Ubuntu, which I use. I have no idea if this thing supports CUDA though, and do not really hold my breath for it.

[danpovey]

I think cmake vs. configure-script plus make is the real choice here. Including improvements to the existing configure scripts.

My experiences with bazel have been terrible, mostly because it's extremely hard to build itself. cmake does seem to be quite popular, e.g. pytorch uses it and I think some of Kaldi's own dependencies require it. I don't have much experience with it though.

On Mon, Mar 11, 2019 at 2:07 PM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

it seems MS has the intention to support CMake in the VS. At least I've heard something for VS2017, not sure if it happened or if they dropped it or what -- @kkm https://github.com/kkm would probably know.

They do. Which does not mean you can open a Kaldi-sized project with it. Since CMake generates about 500 separate project files, one per library and one per exe, it just blows up and dies spectacularly. For smaller projects, like zlib, it worked okay... It's not even MS to blame this time, it's how CMake handles Visual Studio. One output, one separate project in a separate directory.

I do not care much about building the whole rig on Windows. I tried it once, and it was less that exciting in the end. I even ran an eg end to end (tedlium, IIRC), using Cygwin bash. I even added support for it to Kaldi code, so things like popen pipe stuff through bash.

Windows 10 has added two intereting features. One is real functioning symlinks (in fact, they were there in NTFS all the time, but with a twist: you needed to be an admin to create symlinks, and, while you can grant any account a separate privilege to create symlinks, this privilege is unconditionally suppressed for admin accounts, unless you elevate, i. e. "sudo". So, to translate to Linux lingo, you must either be either a non-sudoer or create symlinks under sudo. In W10, all you need is enable developer mode).

The second is native support for Linux user-mode binaries. This is an interesting way to go, but I am not sure if this will cut it for Kaldi. I'll try at some point. It's an interesting Linux. It actually runs in a kind of container, with a special init. Everything in else in the usermode is just a normal Linux, with ELF binaries running out of the box. There are a few distros available, including Ubuntu, which I use. I have no idea if this thing supports CUDA though, and do not really hold my breath for it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471655863, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu_AjSGhQsqWihxqDo9ZSv_IfRpwJks5vVptUgaJpZM4bnduF .

[galv]

I have personally enjoyed using CMake quite a lot. It is a fairly involved build system to set up, but it is also easiest to extend in an already existing C++ project, imo.

It also has the best CUDA support. Instead of our configure script which goes through a bunch of weird places that all the BLAS variants can be installed, you can just do find(BLAS) (or something like that). And you no longer need make depend! Halelujah.

It even supports building pip wheels for python projects with C++ dependencies easily via scikit-build https://github.com/scikit-build/scikit-build. I've done it myself here. It requires very few lines of code. The setup.py https://github.com/galv/galvASR/blob/dev/setup.pyfile is small. And the CMakeLists.txt file basically just needs a single line to build a shared object for python to load: https://github.com/galv/galvASR/blob/dev/galvASR/tensorflow_ext/CMakeLists.txt#L15 https://github.com/galv/galvASR/blob/dev/galvASR/tensorflow_ext/CMakeLists.txt#L15

No idea about cmake's support for windows. Frankly, it has never been a big priority for me. I would say that is true for a lot of the ML/scientific computing community.

On Mon, Mar 11, 2019 at 11:37 AM Daniel Povey notifications@github.com wrote:

I think cmake vs. configure-script plus make is the real choice here. Including improvements to the existing configure scripts.

My experiences with bazel have been terrible, mostly because it's extremely hard to build itself. cmake does seem to be quite popular, e.g. pytorch uses it and I think some of Kaldi's own dependencies require it. I don't have much experience with it though.

On Mon, Mar 11, 2019 at 2:07 PM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

it seems MS has the intention to support CMake in the VS. At least I've heard something for VS2017, not sure if it happened or if they dropped it or what -- @kkm https://github.com/kkm would probably know.

They do. Which does not mean you can open a Kaldi-sized project with it. Since CMake generates about 500 separate project files, one per library and one per exe, it just blows up and dies spectacularly. For smaller projects, like zlib, it worked okay... It's not even MS to blame this time, it's how CMake handles Visual Studio. One output, one separate project in a separate directory.

I do not care much about building the whole rig on Windows. I tried it once, and it was less that exciting in the end. I even ran an eg end to end (tedlium, IIRC), using Cygwin bash. I even added support for it to Kaldi code, so things like popen pipe stuff through bash.

Windows 10 has added two intereting features. One is real functioning symlinks (in fact, they were there in NTFS all the time, but with a twist: you needed to be an admin to create symlinks, and, while you can grant any account a separate privilege to create symlinks, this privilege is unconditionally suppressed for admin accounts, unless you elevate, i. e. "sudo". So, to translate to Linux lingo, you must either be either a non-sudoer or create symlinks under sudo. In W10, all you need is enable developer mode).

The second is native support for Linux user-mode binaries. This is an interesting way to go, but I am not sure if this will cut it for Kaldi. I'll try at some point. It's an interesting Linux. It actually runs in a kind of container, with a special init. Everything in else in the usermode is just a normal Linux, with ELF binaries running out of the box. There are a few distros available, including Ubuntu, which I use. I have no idea if this thing supports CUDA though, and do not really hold my breath for it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471655863, or mute the thread < https://github.com/notifications/unsubscribe-auth/ADJVu_AjSGhQsqWihxqDo9ZSv_IfRpwJks5vVptUgaJpZM4bnduF

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471667364, or mute the thread https://github.com/notifications/unsubscribe-auth/AEi_UJTXXzvOjGKKCPx3s_Oi2JT3WAbiks5vVqJNgaJpZM4bnduF .

-- Daniel Galvez http://danielgalvez.me https://github.com/galv

[galv]

I would also like to mention that I personally like how cmakemakes installing dependencies, really, really easy.

For example, even though wav2letter and flashlight have a nightmarish number of dependencies, it is actually fairly easy to install them via cmake because almost all of the dependencies are cmake projects. I made an example project doing that here: https://github.com/galv/wav2letter-sample-project/blob/master/CMakeLists.txt I probably should have showed that to the FAIR people at some point, since their installation instructions are painful and use too much Docker, but oh well.

On Mon, Mar 11, 2019 at 4:06 PM Daniel Galvez dt.galvez@gmail.com wrote:

I have personally enjoyed using CMake quite a lot. It is a fairly involved build system to set up, but it is also easiest to extend in an already existing C++ project, imo.

It also has the best CUDA support. Instead of our configure script which goes through a bunch of weird places that all the BLAS variants can be installed, you can just do find(BLAS) (or something like that). And you no longer need make depend! Halelujah.

It even supports building pip wheels for python projects with C++ dependencies easily via scikit-build https://github.com/scikit-build/scikit-build. I've done it myself here. It requires very few lines of code. The setup.py https://github.com/galv/galvASR/blob/dev/setup.pyfile is small. And the CMakeLists.txt file basically just needs a single line to build a shared object for python to load: https://github.com/galv/galvASR/blob/dev/galvASR/tensorflow_ext/CMakeLists.txt#L15 https://github.com/galv/galvASR/blob/dev/galvASR/tensorflow_ext/CMakeLists.txt#L15

No idea about cmake's support for windows. Frankly, it has never been a big priority for me. I would say that is true for a lot of the ML/scientific computing community.

On Mon, Mar 11, 2019 at 11:37 AM Daniel Povey notifications@github.com wrote:

I think cmake vs. configure-script plus make is the real choice here. Including improvements to the existing configure scripts.

My experiences with bazel have been terrible, mostly because it's extremely hard to build itself. cmake does seem to be quite popular, e.g. pytorch uses it and I think some of Kaldi's own dependencies require it. I don't have much experience with it though.

On Mon, Mar 11, 2019 at 2:07 PM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

it seems MS has the intention to support CMake in the VS. At least I've heard something for VS2017, not sure if it happened or if they dropped it or what -- @kkm https://github.com/kkm would probably know.

They do. Which does not mean you can open a Kaldi-sized project with it. Since CMake generates about 500 separate project files, one per library and one per exe, it just blows up and dies spectacularly. For smaller projects, like zlib, it worked okay... It's not even MS to blame this time, it's how CMake handles Visual Studio. One output, one separate project in a separate directory.

I do not care much about building the whole rig on Windows. I tried it once, and it was less that exciting in the end. I even ran an eg end to end (tedlium, IIRC), using Cygwin bash. I even added support for it to Kaldi code, so things like popen pipe stuff through bash.

Windows 10 has added two intereting features. One is real functioning symlinks (in fact, they were there in NTFS all the time, but with a twist: you needed to be an admin to create symlinks, and, while you can grant any account a separate privilege to create symlinks, this privilege is unconditionally suppressed for admin accounts, unless you elevate, i. e. "sudo". So, to translate to Linux lingo, you must either be either a non-sudoer or create symlinks under sudo. In W10, all you need is enable developer mode).

The second is native support for Linux user-mode binaries. This is an interesting way to go, but I am not sure if this will cut it for Kaldi. I'll try at some point. It's an interesting Linux. It actually runs in a kind of container, with a special init. Everything in else in the usermode is just a normal Linux, with ELF binaries running out of the box. There are a few distros available, including Ubuntu, which I use. I have no idea if this thing supports CUDA though, and do not really hold my breath for it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471655863 , or mute the thread < https://github.com/notifications/unsubscribe-auth/ADJVu_AjSGhQsqWihxqDo9ZSv_IfRpwJks5vVptUgaJpZM4bnduF

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471667364, or mute the thread https://github.com/notifications/unsubscribe-auth/AEi_UJTXXzvOjGKKCPx3s_Oi2JT3WAbiks5vVqJNgaJpZM4bnduF .

-- Daniel Galvez http://danielgalvez.me https://github.com/galv

-- Daniel Galvez http://danielgalvez.me https://github.com/galv

[btiplitz]

The windows cmake might be weak on how it does what it does, but if the input to me is the same, I don't see why I care what cmake does behind the scenes. The current process within kaldi has to be maintained and with cmake, this goes down considerably.

Part of make is pretty simple and that part is very good, but right now the configure script has to be maintained and it seems to me that cmake is better than this scenario. As far as what google puts out, they tend to release a tool and never update it for a decade or more. Mostly unsupported software. Also CMake has many Find..... scripts that already exist.

I'd not over-engineer the build system. And we use cmake here, so I've used it a bunch. There are absolutely things I like and things I don't. I'd take a simple makefile any day, but you don't start with that

[danpovey]

pybind11 has good integration with cmake too, if you are using cmake it makes it quite easy to generate the python package or whetever it is, I think.

On Mon, Mar 11, 2019 at 8:39 PM Brett Tiplitz notifications@github.com wrote:

The windows cmake might be weak on how it does what it does, but if the input to me is the same, I don't see why I care what cmake does behind the scenes. The current process within kaldi has to be maintained and with cmake, this goes down considerably.

Part of make is pretty simple and that part is very good, but right now the configure script has to be maintained and it seems to me that cmake is better than this scenario. As far as what google puts out, they tend to release a tool and never update it for a decade or more. Mostly unsupported software. Also CMake has many Find..... scripts that already exist.

I'd not over-engineer the build system. And we use cmake here, so I've used it a bunch. There are absolutely things I like and things I don't. I'd take a simple makefile any day, but you don't start with that

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471798100, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu8ncQTfMAmlRFm9qkJi3qyaXwZThks5vVvdPgaJpZM4bnduF .

[vdp]

My experience with CMake has also been positive so far, even though I've only used it for relatively small projects. It also has good support for cross-compilation via the so called toolchain files. IIRC Google's Android build system(based on Gradle) is using CMake behind the scenes to build the JNI code.

TLDR; +1 for CMake.

[kkm000]

Looks like there is pretty much a consensus on CMake. : )

Speaking of Windows, if you're up to running a full training pipeline, CMake is going to be the least of your problems anyway, so I am not worrying much about its poor Windows support.

(image source)

[hhadian]

My experiences with cmake have been generally positive too. Recently I had some trouble configuring a project to make shared libraries (instead of static) but that's probably because I have never read any cmake manuals/tutorials. So I think cmake might be good.

However, the current build system is already very good in the sense that it works on most machines without any issues. So unless we plan to add a lot of more dependencies to Kaldi, I'm not sure what would be the benefit of switching to cmake.

[danpovey]

We can assess whether cmake is actually better once someone actually comes up with a proposed build system based on cmake. I am hoping someone will volunteer for that. One advantage is that if we do python wrapping with pybind, cmake makes that super easy I believe. And it should be less effort to maintain when things like new Debian or Red Hat versions come out, as we are piggybacking off cmake's work.

On Tue, Mar 12, 2019 at 9:06 AM Hossein Hadian notifications@github.com wrote:

My experiences with cmake have been generally positive too. Recently I had some trouble configuring a project to make shared libraries (instead of static) but that's probably because I have never read any cmake manuals/tutorials. So I think cmake might be good.

However, the current build system is already very good in the sense that it works on most machines without any issues. So unless we plan to add a lot of more dependencies to Kaldi, I'm not sure what would be the benefit of switching to cmake.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471992434, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu2wsEc6fizPH6elL-SXRLnIqsZ8Dks5vV6ZzgaJpZM4bnduF .

[galv]

@danpovey I will volunteer for it. I have the experience it takes. I'm not entirely sure about all of the work involved, but I hope to essentially eradicate the tools directory, replacing it with a single .cmake file which downloads and builds the dependencies for us, as well as converting the src/ directory to use CMakeLists.txt.

In case I get unresponsive, feel free to ping me. If anyone else would like to do this as well, you can also ping me.

I am working on top of your kaldi10 branch, Dan.

On Tue, Mar 12, 2019 at 10:11 AM Daniel Povey notifications@github.com wrote:

We can assess whether cmake is actually better once someone actually comes up with a proposed build system based on cmake. I am hoping someone will volunteer for that. One advantage is that if we do python wrapping with pybind, cmake makes that super easy I believe. And it should be less effort to maintain when things like new Debian or Red Hat versions come out, as we are piggybacking off cmake's work.

On Tue, Mar 12, 2019 at 9:06 AM Hossein Hadian notifications@github.com wrote:

My experiences with cmake have been generally positive too. Recently I had some trouble configuring a project to make shared libraries (instead of static) but that's probably because I have never read any cmake manuals/tutorials. So I think cmake might be good.

However, the current build system is already very good in the sense that it works on most machines without any issues. So unless we plan to add a lot of more dependencies to Kaldi, I'm not sure what would be the benefit of switching to cmake.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471992434, or mute the thread < https://github.com/notifications/unsubscribe-auth/ADJVu2wsEc6fizPH6elL-SXRLnIqsZ8Dks5vV6ZzgaJpZM4bnduF

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-472093991, or mute the thread https://github.com/notifications/unsubscribe-auth/AEi_UBw-iZxvdrbqr1mw59PtmCmncXcrks5vV9-jgaJpZM4bnduF .

-- Daniel Galvez http://danielgalvez.me https://github.com/galv

[danpovey]

Great. Regarding the 'tools' directory, bear in mind that it contains two types of things: (1) libraries and headers that are required to build Kaldi (2) tools, some mandatory (OpenFst), some optional, that are required to run some Kaldi recipes, and ways to install them. For (1), it may make sense to get cmake more involved; but for (2) we still need the tools/ directory.

Also there may be reasons to avoid the "system" versions of, say, OpenFst; for example, it might not be built with the flags that we need it to be built with, or something like that. I guess what I'm saying is: don't be too aggressive, and remember that the convenience of users is paramount; if there is a choice between ease of installation (or being more robust), vs. doing things the "cmake" way, I don't want to do things the "cmake" way.

The check_dependencies.sh script may still be quite helpful because it is quite explicit about how to fix problems. My experience with cmake errors is that while, to me, it tends to be pretty obvious how to address them, they are definitely not obvious to all of the kinds of people who ask questions on the Kaldi lists.

On Tue, Mar 12, 2019 at 1:43 PM Daniel Galvez notifications@github.com wrote:

@danpovey I will volunteer for it. I have the experience it takes. I'm not entirely sure about all of the work involved, but I hope to essentially eradicate the tools directory, replacing it with a single .cmake file which downloads and builds the dependencies for us, as well as converting the src/ directory to use CMakeLists.txt.

In case I get unresponsive, feel free to ping me. If anyone else would like to do this as well, you can also ping me.

I am working on top of your kaldi10 branch, Dan.

On Tue, Mar 12, 2019 at 10:11 AM Daniel Povey notifications@github.com wrote:

We can assess whether cmake is actually better once someone actually comes up with a proposed build system based on cmake. I am hoping someone will volunteer for that. One advantage is that if we do python wrapping with pybind, cmake makes that super easy I believe. And it should be less effort to maintain when things like new Debian or Red Hat versions come out, as we are piggybacking off cmake's work.

On Tue, Mar 12, 2019 at 9:06 AM Hossein Hadian <notifications@github.com

wrote:

My experiences with cmake have been generally positive too. Recently I had some trouble configuring a project to make shared libraries (instead of static) but that's probably because I have never read any cmake manuals/tutorials. So I think cmake might be good.

However, the current build system is already very good in the sense that it works on most machines without any issues. So unless we plan to add a lot of more dependencies to Kaldi, I'm not sure what would be the benefit of switching to cmake.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-471992434 , or mute the thread <

https://github.com/notifications/unsubscribe-auth/ADJVu2wsEc6fizPH6elL-SXRLnIqsZ8Dks5vV6ZzgaJpZM4bnduF

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-472093991, or mute the thread < https://github.com/notifications/unsubscribe-auth/AEi_UBw-iZxvdrbqr1mw59PtmCmncXcrks5vV9-jgaJpZM4bnduF

.

-- Daniel Galvez http://danielgalvez.me https://github.com/galv

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-472107805, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu9eDw0-ZYzsOMdasar5YTbybPlVAks5vV-ddgaJpZM4bnduF .

[galv]

I see. What I like about CMake is that you can add external projects, even those that don't use cmake, like openfst. You can dig around in this file to see what I mean: https://github.com/galv/galvASR/blob/7d5d7826805cbbd0b40954f7eec262f0a7e35f01/galvASR/cmake/external.cmake#L49 It can even download archives and unzip them for installation as well, like we do manually.

So you could imagine removing the tools/Makefile target and replacing it with a new cmake target which encodes the same thing. I'll see if it is reasonably clean to do or not.

(My experience mucking with these details and my unwavering commitment to C++ in spite of it is why I am volunteering for this. I'm not sure how many people have the patience for all the details of C++ build systems!)

Using C++ system libraries rather than project-local libraries which you build yourself is just an exercise in futility nowadays unless you're at a big company which can afford to custom tailor its build, especially after libstdc++ broke the std::string and std::list ABI to be C++11 compatible.

[btiplitz]

@galv I should be able to help you test some of the variants including windows.

[ttroy50]

IMO CMake is the best of the current c++ build systems. It is widely supported, almost all developers know how to use it and it has good support for third party libraries and also a number of ways of finding third party libraries.

If you do go down the CMake route it might also be worth looking at conan package manager as an option for the configuration of the third party libraries / tools.

[cloudhan]

+1 for CMake

It's pain in the ass when you want to integrate a makefile project with a CMake one.

With cmake, you can use vcpkg to handle your dependencies on windows.

[kkm000]

@ttroy50, @cloudhan

Guys, I hate to be a party-crasher, but do not get too gung-ho about package management of CMake, or conan or vcpkg or any package manager whatsoever. Keep in mind that the obligatory Kaldi dependencies are CUDA, MKL and OpenFST, which aren't there. Minor dependencies, maybe.

[ttroy50]

@kkm000 Good point. I'm not sure about CUDA but it should be possible to package OpenFST.

And from a technical point of view I think MLK would be possible too but it might not be allowed because of licensing.

[btiplitz]

I don't think packaging all the dependencies is a good idea. It might be necessary to run some pre-script that downloads the required packages like fst. I am pretty sure this could be done within CMAKE with an execute_process command either calling a script or putting the actual get commands in. And putting the MKL into the pull would be kind of crazy. I've got multiple copies of kaldi and this would balloon my storage requirements to a very crazy amount of storage. The MKL is expected to be installed in specific locations and within a source tree is not a reasonable location for so many reasons. CUDA is installed with an RPM or other package management system. There is no reason for this to change

[MantisClone]

If you intend to implement a CMake build system for Kaldi, I thoroughly recommend taking 2 or 3 days to read Professional CMake: A Practical Guide by Craig Scott. It's the best resource I've found for learning what CMake is capable of, and what you should and should not do.

For downloading dependencies, CMake offers several options:

1. FetchContent downloads a dependency at config time. Works best if the dependency is source code that can be built by CMake. Also works if the dependency is a pre-compiled binary.
2. ExternalProject_Add for downloading a dependency at build time. Works well for all types of dependencies but often implies that you use a "Superbuild" CMake structure. Allows you to define your own Config, Build, Install, and other custom steps if dependency doesn't use CMake. When I say "all types" I mean:
   • Source that builds using CMake
   • Source that builds using some other build system
   • Pre-compiled binaries
   • If a dependency is installed using a package manager then it may require administrator privilege and should be installed manually before initiating the CMake build. Or perhaps you could use (4) or (5) below to invoke the package manager at config time.
3. You can use add_custom_process to run a script at build time. This custom process can then be wrapped in a CMake target using add_custom_target so other targets can depend on the outputs of the script.
4. You can use include(script.cmake) to run a CMake script at config time, using the same variable scope as the rest of your project.
5. Finally, as @btiplitz mentioned, you can use execute_process to run a script at config time.

I try to avoid native scripts like bash and bat because they break cross-platform compatibility and create duplicate work. Using Python or Perl scripts is okay if you're okay with adding them as a dependency. I prefer to write cross-platform CMake scripts and invoke them using the ${CMAKE_COMMAND} -P script.cmake calling syntax.

For creating CMake targets once your dependencies are downloaded, CMake offers several options:

1. Ideally, your dependency provides a config module DependencyConfig.cmake that defines CMake targets for all of its build outputs along with the transitive dependencies between them. It's possible for them to provide a CMake config module even if they do not build using CMake.
2. If your dependency does not provide a config module, then you can write your own CMake find module FindDependency.cmake that attempts to find the necessary build outputs, wrap them in CMake targets, and define the transitive dependencies between them. Find modules can work with dependencies that are installed via package manager as well.

Once targets are defined, linking against a dependency is as simple as adding it to your target_link_libraries() list.

EDIT: spelling

[danpovey]

I am still torn on whether to go the CMake route. I guess I feel that just refactoring the 'configure' script (e.g. having it call other scripts that are easier to read) might also be a viable option, as it would be so much more self-explanatory than a CMake setup, and easier to modify. I mean, CMake is just so much framework.

On Tue, Mar 26, 2019 at 10:38 AM David Hunt-Mateo notifications@github.com wrote:

If you intend to implement a CMake build system for Kaldi, I thoroughly recommend taking 2 or 3 days to read Professional CMake: A Practical Guide https://crascit.com/professional-cmake/ by Craig Scott. It's the best resource I've found for learning what CMake is capable of, and what you should and should not do.

For downloading dependencies, CMake offers several options:

1. FetchContent downloads a dependency at config time. Works best if the dependency is source code that can be built by CMake. Also works if the dependency is a pre-compiled binary.
2. ExternalProject_Add for downloading a dependency at build time. Works well for all types of dependencies but often implies that you use a "Superbuild" CMake structure. Allows you to define your own Config, Build, Install, and other custom steps if dependency doesn't use CMake. When I say "all types" I mean:
   • Source that builds using CMake
   • Source that builds using some other build system
   • Pre-compiled binaries
   • If a dependency is installed using a package manager then it may require administrator privilege and should be installed manually before initiating the CMake build. Or perhaps you could use (4) or (5) below to invoke the package manager at config time.
3. You can use add_custom_process to run a script at build time. This custom process can then be wrapped in a CMake target using add_custom_target so other targets can depend on the outputs of the script.
4. You can use include(script.cmake) to run a CMake script at config time, using the same variable scope as the rest of your project.
5. Finally as @btiplitz https://github.com/btiplitz mentioned, you can use execute_process to run a script at config time.

I try to avoid native scripts like bash and bat because they break cross-platform compatibility and create dupliate work. Using Python or Perl scripts is okay if you're okay with adding them as a dependency. I prefer to write cross-platform CMake scripts and invoke them using the ${CMAKE_COMMAND} -P script.cmake calling syntax.

For creating CMake targets once your dependencies are downloaded, CMake offers several options:

1. Ideally, your dependency provides a config module DependencyConfig.cmake that defines CMake targets for all of its build outputs along with the transitive dependencies between them. It's possible for them to provide a CMake config module even if they do not build using CMake.
2. If your dependency does not provide a config module, then you can write your own CMake find module FindDependency.cmake that attempts to find the necessary build outputs, wrap them in CMake targets, and define the transitive dependencies between them. Find modules can work with dependencies that are installed via package manager as well.

Once targets are defined, linking against a dependency is as simple as adding it to your target_link_libraries() list.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-476676196, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVuyy8JPY8F3B8QGFbKofSv2x2TKc5ks5vajDvgaJpZM4bnduF .

[galv]

@DMats Hi, I haven't met you before, but thank you for recommending that book! I wasn't aware that an authoritative book had actually been written on cmake. I had picked up most of my knowledge from presentations.

I was personally planning to required people to depend on typing make in the tools/ directory for now. The reason why is that if you use ExternalProject_Add, and ever do a make clean in your build directory, you will have to rebuild your dependencies from scratch. And rebuilding openfst is slow.

Regarding packing openfst and then doing Find(OpenFST), no, I don't think that's a good idea. The ABI compatibility is too complicated. We can build it already ourselves, and create an "imported target" in cmake which provides its include directories and libraries for us. I did this in a local change on my PR #3100, but I need to rebase that on top of the latest kaldi10 and push and I need to run.

[galv]

I've been writing up a list of nice things that cmake will give us.

They are:

• Easy ARM Cross-compilation and simulation (You can run the tests in an ARM simulator via a switch to cmake's ctest tool). I personally think this one is very cool.
• It is easy to create a python package with native dependencies via cmake, via scikit-build (I've used it before, and it is professional software, not some hack). I would say this is a very big deal.
• Export Kaldi as a dependency to other cmake projects, since everyone else is using cmake anyway :P We can tighten up dependencies via INTERFACE and non-INTERFACE variables (the former come from PUBLIC dependencies, the latter from PRIVATE) in cmake. For example, in my current cmake-based build PR, I made sure that our build system won't have to expose the macros HAVE_CBLAS,HAVE_OPENBLAS, HAVE_MKL, to any downstream dependers upon the matrix library.
• Easy to do tests with valgrind and cuda-memcheck via adding command line flags when running our tests. No more writing custom rules.
• Possibly faster builds with ninja. CMake can generate ninja build rules automatically for us, just as easily as Makefile rules. We would have to verify that, though.

[MantisClone]

@galv Pleasure to make your acquaintance.

Respectfully, I wasn't trying to suggest that using pre-built OpenFST binaries is the best idea in Kaldi's case.

My FindKaldi.cmake find module finds OpenFST from the tools/ directory after Kaldi's build system has built it. It then exposes OpenFST as an IMPORTED CMake target Kaldi::Kaldi_OpenFst and as a set of variables Kaldi_OpenFst_INCLUDE_DIRS, Kaldi_OpenFst_LIBRARIES. Prefer to use the target.

Obviously, this find module is incomplete as it completely neglects the other build products OpenFST produces. But they could be found in effectively the same way.

#=================================
# OpenFst
#=================================

find_path(Kaldi_Tools_DIR
    NAMES extras/check_dependencies.sh
    HINTS ${Kaldi_ROOT_DIR}/tools
)

find_path(Kaldi_OpenFst_INCLUDE_DIR
    NAMES fst/fstlib.h
    HINTS ${Kaldi_Tools_DIR}/openfst/include
)

find_library(Kaldi_OpenFst_LIBRARY
    NAMES fst
    HINTS ${Kaldi_Tools_DIR}/openfst/lib
)

include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(Kaldi_OpenFst
    REQUIRED_VARS Kaldi_OpenFst_INCLUDE_DIR Kaldi_OpenFst_LIBRARY
    VERSION_VAR Kaldi_OpenFst_VERSION
)
mark_as_advanced(Kaldi_OpenFst_INCLUDE_DIR Kaldi_OpenFst_LIBRARY Kaldi_OpenFst_VERSION)

if(Kaldi_OpenFst_FOUND)
    set(Kaldi_OpenFst_INCLUDE_DIRS ${Kaldi_OpenFst_INCLUDE_DIR})
    set(Kaldi_OpenFst_LIBRARIES ${Kaldi_OpenFst_LIBRARY})
endif()

if(Kaldi_OpenFst_FOUND AND NOT TARGET Kaldi::Kaldi_OpenFst)
    add_library(Kaldi::Kaldi_OpenFst UNKNOWN IMPORTED)
    set_target_properties(Kaldi::Kaldi_OpenFst PROPERTIES
        IMPORTED_LINK_INTERFACE_LANGUAGES CXX
        IMPORTED_LOCATION ${Kaldi_OpenFst_LIBRARIES}
        INTERFACE_INCLUDE_DIRECTORIES ${Kaldi_OpenFst_INCLUDE_DIRS}
    )
    target_link_libraries(Kaldi::Kaldi_OpenFst
        INTERFACE
            ${CMAKE_DL_LIBS}
    )
endif()

EDIT: Clarify that resulting OpenFST target is imported.

[btiplitz]

@danpovey I'd agree that if it takes 2-3 days to understand cmake, then that seems way too complicated, but if 1 person does the transition to cmake, then the 2-3 days does not apply to every person using kaldi as only the person doing the whole conversion needs to understand it. Once converted, once it's working, it would be rare that you would need to understand everything. google search is pretty effective on doing a single task and more effective than scanning any book.

[jtrmal]

I think it looks scary because you guys started to overengineer things without thinking about the way all in Kaldi is generally designed -- not too many bells and whistles and many-layer-frameworks with many different command line switches because you can always change it in the source codes. y.

On Tue, Mar 26, 2019 at 1:41 PM Brett Tiplitz notifications@github.com wrote:

@danpovey https://github.com/danpovey I'd agree that if it takes 2-3 days to understand cmake, then that seems way too complicated, but if 1 person does the transition to cmake, then the 2-3 days does not apply to every person using kaldi as only the person doing the whole conversion needs to understand it. Once converted, once it's working, it would be rare that you would need to understand everything. google search is pretty effective on doing a single task and more effective than scanning any book.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-476766660, or mute the thread https://github.com/notifications/unsubscribe-auth/AKisX_D4lOqt878WWH-02Pyx4k8kAIFTks5valu_gaJpZM4bnduF .

[jtrmal]

so you are already overengineering something that does not exist yet. y.

On Tue, Mar 26, 2019 at 1:47 PM Jan Trmal jtrmal@gmail.com wrote:

I think it looks scary because you guys started to overengineer things without thinking about the way all in Kaldi is generally designed -- not too many bells and whistles and many-layer-frameworks with many different command line switches because you can always change it in the source codes. y.

On Tue, Mar 26, 2019 at 1:41 PM Brett Tiplitz notifications@github.com wrote:

@danpovey https://github.com/danpovey I'd agree that if it takes 2-3 days to understand cmake, then that seems way too complicated, but if 1 person does the transition to cmake, then the 2-3 days does not apply to every person using kaldi as only the person doing the whole conversion needs to understand it. Once converted, once it's working, it would be rare that you would need to understand everything. google search is pretty effective on doing a single task and more effective than scanning any book.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-476766660, or mute the thread https://github.com/notifications/unsubscribe-auth/AKisX_D4lOqt878WWH-02Pyx4k8kAIFTks5valu_gaJpZM4bnduF .

[MantisClone]

I'm a huge proponent of CMake for new projects. But if your existing build system provides all the features you need then I tend to agree with @hhadian that it's not worth switching.

I also agree with @btiplitz that the CMake build maintainer should have a deeper understanding of CMake, but that a well-architected CMake build system can be trivially easy to use.

I think CMake is the most expressive C++ build system available and if done properly, offers a wide variety of benefits while also being less verbose than the equivalent configure script and Makefile.

CMake Benefits:

1. Out of source builds
2. Incremental builds
3. Fast and parallel builds (using Ninja)
4. Multi-language support (C, C++, CUDA)
5. Cross-platform support
6. Cross-compilation support
7. Optimization support (Debug, Release, RelWithDebInfo, MinSizRel)
8. Precise definition of transitive dependencies
9. Dynamic and static linking support
10. Multi-IDE support (Maintainers can use whatever IDE they prefer)
11. Installation from source
12. Redirectable installation
13. Packaging support
14. And all the test/binding tool integrations @galv mentioned in his previous comment

[kkm000]

*** @kkm000 changed this chat topic from "CMake is da coolestest!!!" to "What do we want from our future build system?"

I do not understand all the issues involved in e. g. adding PyTorch support to the build process, so I am not leaning one way or another, I simply lack information. If anyone wants to summarize the issues that we currently foresee, that would be a very significant step into evaluating possible build improvements.

I agree that CMake is cool (all software is), or that CMake is a pain (all software is), or that a well architected CMake build system makes it all easy-peasy (but so does a well-architected build system written in shell script alone; the question of the effort involved must be addressed explicitly) . To me, every tool has it strengths and limitations, and we are deciding a gravely important issue here. If we get it wrong now, there will be more build maintenance and less speech science for a while. Let's please stay together on the solid common ground of reason, not inherently nebulous personal passions.

If we are not going to add anything significant to the build, I entirely agree with the Dan's latest comment. Our configure system is custom-built, it maybe unconventional, but it is not broken. We can support it with low effort. It's small enough, and it's transparent, unlike e. g. automake¹. Kaldi codebase is large, but its build process is very regular. Now, remember, CMake sits at the same level of meta-make as automake. Yes, we do have some problems in concurrent make (some build chains run twice; not only on Travis, but I've recently seen it locally), but no meta-layer will help solve them; they in a sense tangential. I am touching on this only to point that while a meta-maker won't help us get rid of old bugs, it will certainly bring on his own.

If we replace our configure with another system, we will probably make our life a bit harder, at the least initially. I want to understand what we'd gain from this trade. Yes, CMake is great compared to automake, no question about that, but I am not at all convinced it has a lot of benefits compared to our other option, which is doing nothing. It's not easy to beat, come think of it.

So I am asking someone who both has a good vision of the project's direction, and wants to create the outline for the rest of us, please do that. It should touch on (1) where we want to be, build-wise, in a few months to a couple years (e. g., build PyTorch libraries, ...), (2) what is involved in this, as this maybe unfamiliar (e. g. “to build PyTorch-compatible DLLs we need to be be able to do X, Y, Z, http://... has a good overview ”), and, optionally, (3) (a) what foreseeable obstacles on the way we must be aware of, and (b) the annoyances we currently casually endure to the point of not even noticing them, but which would still be beneficial to take care of. This would be tremendously helpful, I repeat myself here, in keeping this discussion more focused on our actual wants and needs. Multiple persons contributing input to one or more of the above points are indeed welcome. I'd do that, but I am not qualified, sorry, at the very least w.r.t point (1) above. And (1) is probably the most important point, so it better be exhaustive.

Knowing (1), I'll be happy to help research (2). But for now, need help please!

---

¹ The only script I know of which is even less readable is sendmail.cf; if you do not know what it is, good for you; I strongly encourage you to please stay blissfully ignorant about it.

[galv]

Hi @kkm000 it will take a while to respond fully to your questions. But I will begin with one of my desires for (1) by mentioning that I have a hunch that cmake will actually speed up our build quite a bit. This is because cmake will allow us to use Ninja. Unlike Make, Ninja knows that you can compile a .cc file before all of the .o, .a, and .so files that it links to have been created. I've copy-pasted out individual projects' dependencies below (this is from src/Makefile):

base: base/.depend.mk
matrix: base
util: base matrix
feat: base matrix util gmm transform tree
tree: base util matrix
gmm: base util matrix tree
transform: base util matrix gmm tree
sgmm2: base util matrix gmm tree transform hmm
fstext: base util matrix tree
hmm: base tree matrix util
lm: base util matrix fstext
decoder: base util matrix gmm hmm tree transform lat
lat: base util hmm tree matrix
cudamatrix: base util matrix
nnet: base util hmm tree matrix cudamatrix
nnet2: base util matrix lat gmm hmm tree transform cudamatrix
nnet3: base util matrix lat gmm hmm tree transform cudamatrix chain fstext
rnnlm: base util matrix cudamatrix nnet3 lm hmm
chain: lat hmm tree fstext matrix cudamatrix util base
ivector: base util matrix transform tree gmm
#3)Dependencies for optional parts of Kaldi
onlinebin: base matrix util feat tree gmm transform sgmm2 fstext hmm lm decoder lat cudamatrix nnet nnet2 online
# python-kaldi-decoding: base matrix util feat tree gmm transform sgmm2 fstext hmm decoder lat online
online: decoder gmm transform feat matrix util base lat hmm tree
online2: decoder gmm transform feat matrix util base lat hmm tree ivector cudamatrix nnet2 nnet3 chain
kws: base util hmm tree matrix lat

As you can see, basically everything depends upon base and util, so the sources in those directories must be built before anything else is built.

Also, Dan knows that I've done a lot of surgery-like things to make our build system support various installations of the BLAS libraries, as well as CUDA. It's honestly a big pain for me to have to reason about a bunch of different systems all at once. I remember the one time that I had to create the tools/ATLAS_headers library because. It's pretty cool to know I am one of the few people who knows all kinds of intricacies of Centos/RHEL vs Debian/Ubuntu, but I would rather not be doing that kind of work when I am a machine learning engineer.

Finally, I've experimented a lot with combined kaldi with other frameworks here: https://github.com/galv/galvASR

The biggest thing I've found is that Kaldi is hard to package as a dependency of another library. For example, it used to be that if you depended upon just about any meaningful kaldi header file, you needed to know which version of BLAS you had linked to, and make sure that youse have the (non-namespaced) macro HAVE_ATLAS, HAVE_MKL, or HAVE_OPENBLAS defined correctly. In Modern CMake, a kaldi matrix library linked to OpenBLAS knows that all downstream users of it must define HAVE_OPENBLAS, because it is marked as one of its INTERFACE_COMPILE_DEFINITIONS, which are passed transitively. See that I set that here when OpenBLAS is used: https://github.com/kaldi-asr/kaldi/pull/3100/files#diff-af3b638bc2a3e6c650974192a53c7291R23, and the matrix "target" doesn't even need to know of HAVE_OPENBLAS: https://github.com/kaldi-asr/kaldi/pull/3100/files#diff-e7dffd226ecbe24e17be772c2445893eR5

Another thing: Kaldi does not have its own namespace for being #include'd by other projects. You can choose to set your include path to point to /src, so you can #include "feat/feature-mfcc.h" (or whatever), but what if another project has a directory called feat, containing feature-mfcc.h? It would be much better if we could include another directory between src and feat, called "kaldi", to "namespace" our include paths. My understanding is that CPack allows us to do this.

Also, that work also taught me that basically everyone else is using cmake, except Tensorflow (but bazel is a pain and you can easily package tensorflow as an imported depency in cmake like this anyway: https://github.com/galv/galvASR/blob/7d5d7826805cbbd0b40954f7eec262f0a7e35f01/galvASR/cmake/external.cmake#L131. Note that I wrote that code before I understood modern cmake, so no one should blindly copy-pasta that without using those variables to create an imported library). It's very easy to depend upon other cmake projects in a cmake project.

I know that you also use Kaldi in industry, as opposed to a research tool, Kirill so I am actually pretty curious what kinds of problems you've had with it in this context.

That was a mouthful. Ask me anything. Good night.

[cloudhan]

same as galv, the dependency pain I face mainly in integrating kalid into other project, not kaldi's dependencies, since almost all well organized project is self-contained. Build is fairly easy, thanks to contributors, you just make make make, that's it. But when it comes linking with kaldi, you must maintain some separate build script, built it, properly copy it, some other script to import it. and you must carefully check all build between version update, in case your script doesn't address the incoming changes.

@galv

... before I understood modern cmake ...

then I'll assume you've learnt a well modern cmake, would your recommend some good learning resource?

[galv]

@cloudhan Just read https://crascit.com/professional-cmake/ that @DMats mentioned, and maybe this: https://meetingcpp.com/mcpp/slides/2018/MoreModernCMake.pdf and this: http://on-demand.gputechconf.com/gtc/2017/presentation/S7438-robert-maynard-build-systems-combining-cuda-and-machine-learning.pdf

[kkm000]

@galv, thanks for your thoughts.

I know that you also use Kaldi in industry, as opposed to a research tool, Kirill so I am actually pretty curious what kinds of problems you've had with it in this context.

Since I compile the production code with only one math library (MKL), and only on Windows, I did not face a lot of obstacles in integrating it. I add Kaldi repo as a Git submodule and compile everything Kaldi I need (which is the decoding part) into a single big static library (this may become a problem; I think there is a limit on a library file size, and it's getting close to 1GB). The only part where I had a trouble decoupling train-time code from decode-time is nnet3; almost everything has to go in, but this is not a big deal, in the end. I'm using link-time codegen for optimization (this also deals with dead train-time-only code), and then PGO for release. As for the include path, yes, I'm pointing to src/, and include what I use. If there would have been a conflict, yes, as you point, I'd just rearrange the directory structure and use one level above. (EDIT) Aw, I was typing faster than thinking. If there would have been a conflict, that would have been ugly. But chances of having a file named "feat/feature-mfcc.h" are really negligible, IMO. (/EDIT) Actually, I think that for a library of such a size, having a file that "includes everything" may not be the best idea, as the "everything" is different for different users.

I'm trying to maintain as few changes to the codebase as possible; the way the Kaldi project is receptive to the changes makes this very easy (@danpovey -- thank you and sorry for the pain I caused with the custom logging hook:) )

But given my environment, I'm using MSBuild, and it's way smarter about evaluating dependencies in large projects than make.

I would say I had more trouble with OpenFST than Kaldi, where I need to put some files into the include path before it and override them (lock.h, log.h). With a static sanity check source file like this :)

#include <fst/log.h>
#include <fst/lock.h>

#ifndef FST_LIB_LOG_H__OVERRIDDEN
#error Invalid <fst/log.h> included
#endif
 . . . then same for lock.h ...

There used to be more hacks, but @jtrmal and I now maintain a Windows fork of OpenFST so this has become much simpler.

With Kaldi, I do the same trick only for the file version.h, which is normally autogenerated and then included in "stock" Kaldi sources. But this is using an include path override specific to the project, so it would not possibly conflict with another file called version.h.

The most troublesome part is another set of xplat libraries, which I do not want to name in the "most troublesome" context--and it's actually very good, just how the comparison turns out. Their build is CMake-based, but I am far from trying to build a regression from one data point. :)

I would rather not be doing that kind of work when I am a machine learning engineer.

Do we really have a choice? Maybe I'm getting away too easily because we're using Kaldi in only one server codebase. Also, it certainly helps that Microsoft development tools are really unbeatable even close by anything in the Unixland. But then, LTCG+PGO is an optimization really worth it (Pogo rearranges code based on saved profiling results to minimize branch predictor misses, and for non-matrix code, like the decoder, it's a big thing), and I am not sure if it would be possible for a precompiled DLL. So when it comes to production-level code, I inevitably need to take a break from being an ML engineer. Frankly, I'm not one even on other days, most of my time anyway :)

[kkm000]

By the way, version.h. This is the thing that causes a full rebuild if you pull code.

But if you disable it, then ccache is your best Ninja, speaking of compilation speed.

The primary benefit of ccache is that you do not need to change anything in makefiles. As a secondary benefit, its documentation does not open with a Philosophical overview :-)

[galv]

Just a quick response on version.h:

Does it really cause a recompile of everything? I suppose it causes a relinking of most things, since my guess is that everything is going to have to relink to kaldi-base.a (or whatever we call it), since that is a dependnecy of everything. That's definitely a big annoyance. Thanks for pointing it out. I do want to mention that ccache does not speed up linking, though. If you disable regenerating version.h, my hunch is that users would save a lot of linking time (no longer need to relink presumably every binary...), regardless of ccache usage.

I should also bring up the most modern gcc builds allow you to pass -fuse-ld=gold as a linker flag to speed up linking, since we are passing around tips now.

By the way, were you referring to something in particular when you mention some documentation opening with a "philosophical overview"?

On Tue, Apr 2, 2019 at 10:20 PM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

By the way, version.h. This is the thing that causes a full rebuild if you pull code.

But if you disable it https://github.com/kaldi-asr/kaldi/blob/be019cd1f/tools/extras/travis_script.sh#L77, then ccache is your best Ninja https://github.com/kaldi-asr/kaldi/blob/be019cd1f/.travis.yml#L50, speaking of compilation speed.

The primary benefit of ccache is that you do not need to change anything in makefiles. As a secondary benefit, its documentation does not open with a Philosophical overview :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-479343969, or mute the thread https://github.com/notifications/unsubscribe-auth/AEi_UHeP8kX2FQLIZRb0j3n-TfbMU1x9ks5vdDowgaJpZM4bnduF .

-- Daniel Galvez http://danielgalvez.me https://github.com/galv

[kkm000]

I do not really remember now how much recompilation did version.h cause with ccache. I only recall the difference was substantial.

Is the build speed really a big issue? My understanding is that most users would rebuild Kaldi pretty infrequently. I, for one, have separate directories for experimentation and for Kaldi hacking; I just want the version with which I do tuning to stay unmodified just in case. And anyway, compared with any decent model training time this is really insignificant, even if you'd recompile everything.

As far as packaging goes, again, the complexity grows exponentially, of course. Kaldi has options A, B, OpenB and so on; static and dynamic, static Kaldi with dynamic MKL, and try to count all options. When integrating with packages X, Y, Z, each of which allows also countless options, it will inevitably come to some combinations that are tricky to get going. But I think I'm failing to understand why it is important to support all, or even the majority of this combinatorics. It's usually not so extremely hard when the tools being integrated are more or less simple, but Kaldi + Tensodflow + PyTorch etc. -- I do not think it is realistic to test an integrated thing in all its possible varieties.

I'm not against CMake, I'm trying my best to keep my mind open. So far it seems that many people think CMake can do a lot of things without hitting a snag, or that it has not a single downside to its use. This is worrying me somewhat. I've been doing my stuff for 30+ years, and if I learned some things, then one of them is that build id hard. The system that you use often always seems the best because you've learned a lot of its quirks and peculiarities, but I can say almost without any qualifiers that there is not a build system free of any of these. And yeah, gmake is petty much full of cryptic stuff. But I'm thinking what would we do if something is not going right. CMake generates unreadable makefiles and horrific MSBuild files. The day I need to debug this build... maybe I'd better be on a vacation. Deep in the jungle of Amazon. Withut any mobile network in a 200 mile range.

Both Bazel and MSBuild are good for large projects, but the learning curve is immense. CMake is good, too, when everything goes right. And so on and on. I'm trying to think not so much about the benefits of a particular build system, but rather about the price you pay for the convenience of it. So my pragmatic question about any of them is quite basic: do the benefits (which I do not doubt at all!) outweigh the price you pay for them (these drawback must exists; not a single piece of software it truly perfect,

So, I'm playing a bit of devil's advocate. Above all, I'd like to hear a cost-benefit analizys for each of the systems, as cool-headed impartial, and unbiased as only possible.

were you referring to something in particular when you mention some documentation opening with a "philosophical overview"?

Ah, this: (EDIT: copypasta mishap) https://ninja-build.org/manual.html#_philosophical_overview were you referring to something in particular when you mention some documentation opening with a "philosophical overview". (/EDIT) It's a tongue-in-cheek comment. I was lust much surprised by this turn of events :)

[danpovey]

I think if you configure with dynamic libraries, changing version.h should not cause relinking of everything. We should probably make the default build use dynamic linking.

I do worry, though, about how much slowdown -fPIC causes in decoding.

On Wed, Apr 3, 2019 at 8:54 AM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

I do not really remember now how much recompilation did version.h cause with ccache. I only recall the difference was substantial.

Is the build speed really a big issue? My understanding is that most users would rebuild Kaldi pretty infrequently. I, for one, have separate directories for experimentation and for Kaldi hacking; I just want the version with which I do tuning to stay unmodified just in case. And anyway, compared with any decent model training time this is really insignificant, even if you'd recompile everything.

As far as packaging goes, again, the complexity grows exponentially, of course. Kaldi has options A, B, OpenB and so on; static and dynamic, static Kaldi with dynamic MKL, and try to count all options. When integrating with packages X, Y, Z, each of which allows also countless options, it will inevitably come to some combinations that are tricky to get going. But I think I'm failing to understand why it is important to support all, or even the majority of this combinatorics. It's usually not so extremely hard when the tools being integrated are more or less simple, but Kaldi + Tensodflow

• PyTorch etc. -- I do not think it is realistic to test an integrated thing in all its possible varieties.

I'm not against CMake, I'm trying my best to keep my mind open. So far it seems that many people think CMake can do a lot of things without hitting a snag, or that it has not a single downside to its use. This is worrying me somewhat. I've been doing my stuff for 30+ years, and if I learned some things, then one of them is that build id hard. The system that you use often always seems the best because you've learned a lot of its quirks and peculiarities, but I can say almost without any qualifiers that there is not a build system free of any of these. And yeah, gmake is petty much full of cryptic stuff. But I'm thinking what would we do if something is not going right. CMake generates unreadable makefiles and horrific MSBuild files. The day I need to debug this build... maybe I'd better be on a vacation. Deep in the jungle of Amazon. Withut any mobile network in a 200 mile range.

Both Bazel and MSBuild are good for large projects, but the learning curve is immense. CMake is good, too, when everything goes right. And so on and on. I'm trying to think not so much about the benefits of a particular build system, but rather about the price you pay for the convenience of it. So my pragmatic question about any of them is quite basic: do the benefits (which I do not doubt at all!) outweigh the price you pay for them (these drawback must exists; not a single piece of software it truly perfect,

So, I'm playing a bit of devil's advocate. Above all, I'd like to hear a cost-benefit analizys for each of the systems, as cool-headed impartial, and unbiased as only possible.

were you referring to something in particular when you mention some documentation opening with a "philosophical overview"?

Ah, this: were you referring to something in particular when you mention some documentation opening with a "philosophical overview". It's a tongue-in-cheek comment. I was lust much surprised by this turn of events :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-479476496, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu4w625oOald06FH9iq4rGuBJ9YaPks5vdKSagaJpZM4bnduF .

[kkm000]

@galv: I was already half-asleep, looks like a pasted a piece of your text instead of the link, it made my mumbling unreadable, sorry: https://ninja-build.org/manual.html#_philosophical_overview. I'm just making friendly fun of it, not judging it by the cover. It does not mean Ninja itself is bad, or good, or ugly. I'm afraid to think how the “philosophical overview” of GNU make would look. Like, Nietzsche meets Feyerabend.

@danpovey : Yeah, the lattice walk is the part that I found uber-sensitive to optimizations. And pessimizations, like -fPIC, alike, no wonder. But dynamic linking is faster, indeed. And now you've got me thinking: we have over 9000 configure flags: static this, dynamic that. threaded with TBB, threaded with IOMP... Sometimes I think it would just better be a few sensible presets, and then hack kaldi.mk to the bottom of your soul if you want. The overall scenarios are not that numerous:

• As the "toolkit user /eg developer/data scientist" (an umbrella, but you understand), I would probably just want to say, give me the fastest binaries, I'm OK to compile them once then run over and over. Presumably, this should be the default, as this user is the one least interested in the options, and just wants to ./configure && make. Our cautious -O1 does not cut it.
• As the Kaldi hacker, I'd go with a faster compile and perhaps -Og.
• As the production system integrator... this is such an intimate tweaking, tuning and profiling process, that we can hardly help, except adding API for that, and its perfectly fine.
• Now, I am all of the three above but not this one (but @galv is): As far as integrating Kaldi with other ML toolkits to do data sci^H^H witchcraft, I have no experience. I have a hunch that there is much intersection with the first bullet, i. e. we should be the fastest at runtime at an expense of compile time. Roughly speaking, I would not venture to crunch 10K hours of speech with a packaged OpenBLAS; I'd rather want a build optimized dearly for my CPU. But this is the area where concerns are expressed about the compile time, so I may not be possibly grokking the whole picture. @galv, @cloudhan, please chime in on this.
• Am I missing more use cases?

If I am not missing much, it looks like we should better be the fastest at run time for just ./configure -- and the compilation speed has exactly zero bearing here; it better be faster that not, but not at a zilch of expense of runtime performance! I do not think that switching to .so is in line with that idea, if you accept my reasoning.

Regarding the first bullet, it has been long on my laundry list to experiment with different math optization flags in gcc. Another one is the decoding code. This is where e. g. memory aligning of vectors and swapping inner/outer loops could be beneficial. I hope sincerely I could get to these at some point.

(Side note, the CUDA decoder is a super interesting project, and an opportunity to learn right from the horse's mouth!)

[kkm000]

@galv, thanks for the tip on using gold. I did never pay attention, but I should have. Google has amazing toolsmiths. Internally, so much of 3rd party tools in the chain are replaced gradually and continuously with home-build ones, with mostly similar interface, but better optimizing, faster, catching errors statically &c. &c. This is the team from which asan and msan came from, too, among many others.

[jtrmal]

Gold is both solution to many problems in the world and the reason of the problems. 🤪

On Wed, Apr 3, 2019 at 22:48 kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

@galv https://github.com/galv, thanks for the tip on using gold. I did never pay attention, but I should have. Google has amazing toolsmiths. Internally, so much of 3rd party tools in the chain are replaced gradually and continuously with home-build ones, with mostly similar interface, but better optimizing, faster, catching errors statically &c. &c. This is the team from which asan and msan came from, too, among many others.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-479652772, or mute the thread https://github.com/notifications/unsubscribe-auth/AKisX-ry6nEPCNvIOHLV0I9YCShlFfobks5vdROQgaJpZM4bnduF .

[btiplitz]

Google has a history of developing and abandoning things. Just because google uses it does not make it wise

[btiplitz]

@danpovey https://stackoverflow.com/questions/31332299/why-not-always-use-fpic-position-independent-code and https://gcc.gnu.org/ml/gcc/2004-06/msg01956.html

the 2nd is very old, but says that if compiling 03, the difference is very small. The 1st, which is newer, says the CPU added instructions to make this work very fast. I've not done any performance testing of this, but we do compile all our code with -fPIC and have never worried about the performance. Though I personally do tend to be very concerned with speed, so it's surprising I've never tested the difference.

I do have a performance test, so I guess I could look at this. (like I have so much free time, not really)

Personally I would think if fPIC has any measurable slow down, adding it as a build option would be great for me, but I would think optional would be a very easy option to add

[kkm000]

@jtrmal:

Gold is both solution to many problems in the world

Yeah, it's the funny how we're using the language! I have more background in the symbolic than the connectionist AI, and it's tightly connected to linguistics: syntax, semantics, pragmatics, all this stuff. So I often notice, too. On the same note:

Funny: From the expect(1) manual:

BUGS
       It was really tempting to name the program "sex" (for either "Smart EXec" or "Send-EXpect") [...]

Srsly: I did this to test the new install_mkl.sh on multiple systems using Docker, because the script had to muck with different package managers: https://github.com/kkm000/install_mkl. I put in some time to make it a bit more generic that it likely had to be, using gmake to parallelize Docker builds and expect(1) to test the script interactive prompts. Just keep in mind it exists, in case we need to test something else, e. g. check_dependencies.sh or other scripts that are kind of critical in the early Kaldi bootstrapping, by a large number of Linux distros.

There is not much documentation but the general idea is that make recurses in parallel into the same Makefile with a variable set to encode other variable names that in turn define the test "flavor" (like debian+9), so the Makefile is essentially one large if/else/endif, except for common system definitions. There is no real dependency checking, as docker avoids duplicating work nicely; I used gmake to handle parallel execution, mostly.

---

Which takes me back to the point of this discussion: yeah, make is nasty in both its syntax and behavior, and outdated. But in my experience, the nasty part applies equally to MSBuild, Bazel, CMake (sorry); I have yet to encounter a build system that I'd call sweet. It's a no-brainer to script a build of helloworld.exe from hello.c and world.c in any of these, but as you push toward advanced features, all of them push back at you even harder, Newton's Third law notwithstanding. Maybe it's just the fact of life that build is hard. This is just to explain why I am feeling conservative about redoing the Kaldi's build process from the grounds up: yes, it may be clumsy, but we got it right with a considerable investment of effort. My experience just tells me that constructing any complex build with a good parallelization and correct dependencies and multiple options is not an easy feat, never goes as planned, since you have to play by the restrictive rules of the build system that you might have even been unaware of at the beginning, and the end result may not be as transparent and simple to maintain as it seemed. Enthusiasm is a treacherous friend.

This is why I am calling for a more calm and rational argument and less thinking along the lines of "everyone is on the CMake bandwagon anyway" and "I fell in love with CMake since I first used it." Everyone is different, but personally, I've never had a crush on a certain string of funnily arranged binary digits... :)

I am not against CMake at all, and an it seems like the consensus is it is the best choice-- if we were at a blank slate. But we aren't. The tough question is the true cost and true benefits of rewriting the make process given we've got what we've already got. It may be ugly, but it works. The main exercise is to figure out where it does not work; the cases that make it painful to maintain (@galv, I hear you!); where it just could not be extended with a sane effort to do what we want it to do. I would really like to provide some input in this area, but I maybe too much used to the Kaldi build, so it seems easy, and I'm not using it in many different ways. But I'll think of this, too, and go over this thread carefully once more.

I would not rather start with a proof-of-concept script to show "look, its so easy in CMake!", because POC is just that, and Pareto will inevitably come after you when it's 80% done. It's a big decision and a big commitment of limited time and hands. And build is hard.

[galv]

@btiplitz the gold linker is part of the GNU binutils. The developer of it just happened to work at Google. It's not some kind of Google abandonware.

This discussion has gotten a little bit wonky. I think I need to write a coherent response to @kkm000 at this point, but it will take a bit (sorry!).

[kkm000]

@galv, please don't be sorry, there is no rush. This whole question is just asking for slow, thorough thinking.

[danpovey]

Thanks for the comments (catching up on this thread as I was on a plane yesterday),

RE the configure-script defaults, if the speed change from -fPIC isn't that large (e.g. less than 10%) I'd rather have dynamic by default because making the build slow increases barriers to entry, which matters as much as performance. As long as we make it easy to make it fast.

I'm still not 100% decided on cmake; let's see what Daniel's work ends up looking like.

On Wed, Apr 3, 2019 at 5:28 PM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

@galv https://github.com/galv, please don't be sorry, there is no rush. This whole question is just asking for slow, thorough thinking.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/3086#issuecomment-479705281, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVuzk6lWzVDNSWlDSU_aHdaayn3YFEks5vdUc-gaJpZM4bnduF .