search

tukaani-project / xz

XZ Utils
https://tukaani.org/xz/
Other
623 stars 115 forks source link

5.6.2 release tarball configure script fail to set FILECMD, causing --disable-static ineffective on MIPS64 #142

Closed xry111 closed 1 month ago

xry111 commented 2 months ago

The 5.6.2 release tarball is generated with libtool-2.5.0, where a bug causes it fail to set FILECMD. The fix of the bug is https://git.savannah.gnu.org/cgit/libtool.git/commit/?id=9a4a02615c9e7cbcfd690ed31874822a7d6aaea2. The effect of the bug can be seen in config.log (on all architectures):

configure:9458: checking for file
configure:9479: found /usr/bin/file
configure:9491: result: :

While the bug is "harmless" on most architectures, on MIPS64 it wrecks havoc. There's something like

mips64*-*linux*)
  # Find out what ABI is being produced by ac_compile, and set linker
  # options accordingly.
  echo '[#]line '$LINENO' "configure"' > conftest.$ac_ext
  if AC_TRY_EVAL(ac_compile); then 
    emul=elf
    case `$FILECMD conftest.$ac_objext` in
      *32-bit*)
    emul="${emul}32"
    ;;
      *64-bit*)
    emul="${emul}64"
    ;;   
    esac 
    case `$FILECMD conftest.$ac_objext` in
      *MSB*)
    emul="${emul}btsmip"
    ;;   
      *LSB*)
    emul="${emul}ltsmip"
    ;;   
    esac 
    case `$FILECMD conftest.$ac_objext` in
      *N32*)
    emul="${emul}n32"
    ;;
    esac
    LD="${LD-ld} -m $emul"
  fi

So on MIPS when FILECMD=:, we get only emul=elf. Then as ld -m elf does not work at all, we get:

configure:13624: checking whether the gcc linker (/usr/bin/ld -m elf) supports shared libraries 
configure:14897: result: no

and --disable-static becomes ineffective.

In xz this is the second case the autoconf-based building system contributes to a breakage (the first case is already "well known" or "notorious"). While it's too premature to drop autoconf, I'd suggest not to use a libtool alpha/beta release (2.5.0 is an alpha release) in the future. I.e. just use the latest stable release (2.4.7 as at now).

Larhzu commented 2 months ago

The 5.6.2 release tarball is generated with libtool-2.5.0, where a bug causes it fail to set FILECMD.

Thanks!

On 2024-06-19, I created a patch based on the Libtool upstream commit. The patch is linked from the stable releases section of the home page. It's also described and included in the relevant three releases on GitHub releases page. I also sent an announcement to the xz-announce and xz-devel mailing lists. Links to the upstream commit and Linux distributions mailing list were included too. I mentioned the patch on #tukaani IRC channel as well.

I'm sorry that this information didn't reach you.

While it's too premature to drop autoconf, I'd suggest not to use a libtool alpha/beta release (2.5.0 is an alpha release) in the future. I.e. just use the latest stable release (2.4.7 as at now).

This is a somewhat tricky question. :-(

I have used Libtool from Arch Linux. I have been aware (and happy enough) that it ships with a Git snapshot of Libtool. However, I don't know why Arch Linux has done that. My guess is that historically it helped more than it hurt.

Libtool 2.4.6 was released in 2015. The next and current stable, 2.4.7, was released in 2022. Before 2.4.7, creating packages with a snapshot version of Libtool meant that downstreams didn't need to patch Libtool fixes that had been waiting in Libtool's Git repository for quite a long time. To me it felt like an OK compromise when Libtool upstream had a lack of developer resources, so Arch's packaging style felt fine to me in that situation.

2.4.7 is 2½ years old now. In 2024, Libtool's development has gotten much more active again. 2.5.2 is a beta release and so it sounds that a new stable release is getting close. :-) At this point I think it's better to not go back to 2.4.7 with XZ Utils because doing so would re-introduce Libtool issues that were fixed after the 2.4.7 release. Yes, there could be new bugs in 2.5.2 but there is no perfect solution.

Creating an alternative xz tarball with Libtool 2.4.7 could be an option. Currently I feel it's not worth it.

When 2.6.0 is out, perhaps it would be good to then stick with stable releases. I wonder what Arch will do. In any case, I could build a separate Libtool version to create xz releases.

For what it's worth, long ago, I was once asked to create a xz tarball with a newer Libtool because the version I had used didn't include some required change or fix (it's not easy enough for all users to just regenerate the build system files themselves). Unsurprisingly with Arch's Libtool package, such requests haven't happened. Before this Libtool bug, no one had asked me to downgrade the version used in xz but now there have been two or three of such comments (which I find very understandable).

In xz this is the second case the autoconf-based building system contributes to a breakage (the first case is already "well known" or "notorious").

I haven't attemted to count. With "notorious", I assume you refer to the backdoor trigger that was intentionally inserted to the release tarballs. While Autotools definitely was a contributing factor, it was a special kind of "breakage" still. However, there have been normal build system issues that have broken the build.

For example, the bug that was fixed in 5a5bd7f871818029d5ccbe189f087f591258c294 has some similarities to the recent Libtool issue in sense that one couldn't build a shared liblzma if --disable-threads was used (so --disable-static became ineffective). Funnily enough, once the bug was fixed, it turned out that one user had accidentally relied on the broken behavior and had to fix his usage of configure options.

XZ Utils 5.8.0 might have two release tarball types: The old style with Autotools and new one without. The new one can be built with CMake and possibly also Meson, and won't include many generated files. Translated man pages likely make sense to include still but if one doesn't like the pregenerated files, it's easy to rm -rf po4a/man && ./po4a/update-po before starting the actual build.

For the paranoid people, the CMake build in XZ Utils 5.6.2 was changed so that it keeps working if one starts with rm -rf tests to get rid of all tests and tests files. Then tests obviously won't be available, which likely does more harm than good as the tests are very quick to run and have caught issues in the past. But in any case, the option to remove the tests exists.

CMake support should already be as good as Autotools support in the master branch. The changes to the CMake build were somewhat big and some are tied to the changes to the C code, so the full CMake support likely won't be backported to 5.6.x. In any case, any issues with CMake in the master branch are taken just as seriously as issues with Autotools.

If your use case has no need to use Autotools, testing CMake support in the master branch is valuable. I know people wish for Meson support and it's on its way but I cannot guarantee that it will be as mature as the CMake support when XZ Utils 5.8.0 is released.

xry111 commented 2 months ago

The 5.6.2 release tarball is generated with libtool-2.5.0, where a bug causes it fail to set FILECMD.

Thanks!

On 2024-06-19, I created a patch based on the Libtool upstream commit. The patch is linked from the stable releases section of the home page. It's also described and included in the relevant three releases on GitHub releases page. I also sent an announcement to the xz-announce and xz-devel mailing lists. Links to the upstream commit and Linux distributions mailing list were included too. I mentioned the patch on #tukaani IRC channel as well.

I'm sorry that this information didn't reach you.

Oops I just searched in this repo and found nothing. I should have checked the list...

/ snip /

Creating an alternative xz tarball with Libtool 2.4.7 could be an option. Currently I feel it's not worth it.

I agree it's not worth it. Anyway one can simply work it around by adding FILECMD=file.

/ snip /

If your use case has no need to use Autotools, testing CMake support in the master branch is valuable. I know people wish for Meson support and it's on its way but I cannot guarantee that it will be as mature as the CMake support when XZ Utils 5.8.0 is released.

We (@lfs-book) still need to use autotools because cmake isn't in our base system, and we don't really want to add it (and all of its dependencies, well but maybe one day the kernel gets some core component written in Rust and we'll have to do...), and we get a circular dependency with it (cmake -> libarchive -> liblzma). Switching to meson is easier because we already have it in the base system, but we'd still need to resolve a circular dependency (meson -> python3 -> liblzma): I know how to do it but we'll just stick with autotools as long as it isn't removed, I think.

Larhzu commented 2 months ago

I should have checked the list...

The home page is the best place to start. I liked to inform the lists so that people who haven't hit the problem yet would get the information earlier.

Perhaps I should have created an issue and pinned it on GitHub too. This project might have too many communication channels (I'm not sure how useful the lists are still).

We @.***) still need to use autotools because cmake isn't in our base system, and we don't really want to add it

Makes sense. Easier bootstrapping and better support for old or obscure systems are reasons why Autotools support will stay for now. Bootstrapping is the reason why a few people have said that they prefer building xz with Autotools for now; you aren't the only one.

circular dependency with it (cmake -> libarchive -> liblzma).

CMake source tree has bundled copies of the dependencies so it's possible to build CMake fairly early. One then needs to build a new copy of CMake after all the dependencies are built, so it's a bit annoying but at least possible. I'm sure you knew this though.

Switching to meson is easier because we already have it in the base system, but we'd still need to resolve a circular dependency (meson -> python3 -> liblzma)

muon might help with this as it can bootstrap itself without dependencies. I won't attempt muon compatibility in xz at first though.

xry111 commented 2 months ago

Switching to meson is easier because we already have it in the base system, but we'd still need to resolve a circular dependency (meson -> python3 -> liblzma) muon might help with this as it can bootstrap itself without dependencies. I won't attempt muon compatibility in xz at first though.

This is not necessary (for us), we just need to install meson for the host system and cross-compile xz with it. We only aim to resolve unintended host system contamination, not malicious "trusting-trusts," so we just do cross compilation for the early stage and assume the host is "sane."

I.e. we are not live-bootstrap which is much more strict (and requires much more efforts) than us.

xry111 commented 2 months ago

FYI libtool-2.5.3 is released and it's a stable release (at least its NEWS file says so, hmm why they don't make it 2.6.0...)

Larhzu commented 2 months ago

Great! There aren't many commits after 2.5.2. Different projects have different versioning styles. :-)

It's already flagged as out-of-date in Arch Linux so a package is likely to arrive soon. I'm hoping to get XZ Utils 5.6.3 done in a few days, we'll see.