Very slow compilation under Windows with libz-ng-sys

[xTachyon]

I wasn't quite sure if I should make the issue here or on the flate2 repo, but here we go:

I noticed for a long time that the longest crate to compile on one of my projects is libz-ng-sys that I use through flate2. I do a lot of testing with different configurations for fun, so I end up cargo cleaning and then rebuilding a lot, so it's particularly bad.

I use the libz backend because the pure rust one is significantly slower, and the problem is even worse on debug. With a debug full build on said project, on my machine: windows, libz backend: ~33s windows, rust backend: ~10s wsl linux, libz backend: ~14s

Now, I did a bit of investigation on a separate project that only includes libz-ng-sys, by running both commands manually on an empty folder that the build script usually executes, and:

Creating the initial project

This takes ~19s on my system. A big part of it seems to be checking for unixy headers, and the other big part is from checking simd intrinsics. Most of the headers and features are never found for me. This is a lot of time just for creating a project.

Compiling the sources

This takes ~5s on my system. It is not outrageous compared to the first step, but the compilation is single threaded even if the script tries to pass "--parallel" "20". Unfortunately, that's not enough to enable parallel compilation on MSBuild, and it actually actively hurts it. See this SO thread.

Actually passing the multi processing flags correctly brings this down to ~2s, which is better.

Solutions?

The second step can be fixed fairly easily I think, but the first step is a problem so much bigger than the second. Something in the Windows ecosystem really doesn't play well with CMake, given that the build under Linux is far more reasonable.

Maybe the CMake can be optimized? Maybe ditching CMake completely and just calling cl directly with all the right options? Maybe clang-cl does a better job and we should suggest it to people? Or if the problem is MSBuild, then also have an option for Ninja or something else?

In conclusion, I don't know.

[NobodyXu]

I checked the build.rs, it uses crate cc for compiling libz and uses crate cmake (which invokves cmake cmd) for compiling libz-ng.

I have a look at crate cmake and it seems to disable jobserver on windows which prevents make/ninja from creating too many processes.

It could be that you are using a slow build system, it seems that there's make and bsdmake, which doesn't support jobserver and probably not parallel compilation, mingw-make.

There's also ninja but seems that crate cmake does not auto-detect and use ninja on windows.

[Byron]

Thanks for sharing, and thanks for chiming in @NobodyXu!

by running both commands manually on an empty folder that the build script usually executes, and:

Is there something missing that might matter or is it a way to transition into the next paragraph titles?

I have a look at crate cmake and it seems to disable jobserver on windows which prevents make/ninja from creating too many processes.

If it's true that this issue is mostly caused by the cmake crate, would it be better to post an issue there? I doubt that it's feasible to replace the usage of cmake here with anything else.

[xTachyon]

Is there something missing that might matter or is it a way to transition into the next paragraph titles?

Just a way of transitioning.

The cmake crate might be able to fix the second problem, but I very much doubt it can do anything about the first.

[Byron]

The problem I see is that these build systems are used by the underlying source projects, and thus not under control or custody of this FFI crate. For me, unless I am missing something, there would be nothing to do here, particularly since it seems established that the second problem, cmake (the crate) disabling the job server, also cannot be influenced here (yet).