search

rake-compiler / rake-compiler-dock

Easy to use and reliable cross compiler environment for building Windows, Linux, Mac and JRuby binary gems.
MIT License
78 stars 30 forks source link

pkg-config version in x86-linux and x86_64-linux images taking a long time to resolve re2.pc #121

Closed mudge closed 5 months ago

mudge commented 5 months ago

RE2 2024-06-01 uses more of Abseil which has caused the amount of time spent running pkg-config --libs-only-L --static re2.pc within the rake-compiler-dock images for x86-linux and x86_64-linux to significantly increase.

Both images use CentOS 7's latest version of pkgconfig: 0.27.1 but the latest version available is 0.29.2.

By manually downloading, compiling and installing the latest version of pkg-config on ghcr.io/rake-compiler/rake-compiler-dock-image:1.5.0-mri-x86-linux we can compare the difference:

# /usr/bin/pkg-config --version
0.27.1
# time /usr/bin/pkg-config --libs-only-L --static re2.pc
-L/Users/mudge/Projects/re2/ports/arm64-apple-darwin23/abseil/20240116.2/lib -L/Users/mudge/Projects/re2/ports/arm64-apple-darwin23/libre2/2024-06-01/lib  

real    3m41.007s
user    3m40.990s
sys 0m0.025s
# /usr/local/bin/pkg-config --version
0.29.2
# time /usr/local/bin/pkg-config --libs-only-L --static re2.pc
-L/Users/mudge/Projects/re2/ports/arm64-apple-darwin23/libre2/2024-06-01/lib -L/Users/mudge/Projects/re2/ports/arm64-apple-darwin23/abseil/20240116.2/lib

real    0m0.101s
user    0m0.083s
sys 0m0.018s

Is there a way for us to provide a newer version of pkg-config in the rake-compiler-dock images to work around this slowdown?

mudge commented 5 months ago

If you want to reproduce this:

  1. Check out v2.12.0 of the re2 gem
  2. Run bundle install
  3. Run bundle exec rake compile
  4. Start a Docker container with ports mounted, e.g. from within the re2 clone: docker run -v "$(PWD):/re2" --platform linux/amd64 -it ghcr.io/rake-compiler/rake-compiler-dock-image:1.5.0-mri-x86-linux /bin/bash
  5. Set PKG_CONFIG_PATH inside the Docker container to include both Abseil and RE2, e.g. assuming you mounted the clone in /re2, export PKG_CONFIG_PATH=/re2/ports/arm64-apple-darwin23/abseil/20240116.2/lib/pkgconfig:/re2/ports/arm64-apple-darwin23/libre2/2024-06-01/lib/pkgconfig:$PKG_CONFIG_PATH
  6. Run time pkg-config --libs-only-L --static /re2/ports/arm64-apple-darwin23/libre2/2024-06-01/lib/pkgconfig/re2.pc
flavorjones commented 5 months ago

Fascinatin, that's an incredible speedup for sure. I know that 0.28 and 0.29 addressed some recursive path issues, so I'll assume that's why it's faster.

Seems like we've got two options here ...

  1. on the manylinux_2014 images, try to upgrade pkg-config to a modern version
  2. drop manylinux_2014 in favor of a newer version which presumably will have a modern pkg-config

I was planning on doing (2) soon anyway, since the 2014 series is EOL as of 2024-06-30 (later this month!). See https://github.com/pypa/manylinux for details, and #122 for the issue describing this change in more detail.

Let me mess around with (1), though, since I could get that into a patch release on the 1.5.x branch, while (2) would really require a minor release.

flavorjones commented 5 months ago

Possible approach to (1) above is in https://github.com/rake-compiler/rake-compiler-dock/pull/123

flavorjones commented 5 months ago

Closed in #123, will ship v1.5.1 promptly.