Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.99k stars 557 forks source link

./configure mis-detects "ar" and hardcodes missing "ar" in Config_heavy.pl #17791

Open kentfredric opened 4 years ago

kentfredric commented 4 years ago

Description We have developers on Gentoo testing some unusual setups, particularly:

Under this configuration, perl ./configure fails to detect a correct ar, and later, embeds ar = ar in Config_heavy.pl

However, this failure does not stop perl from compiling, but it does make ExtUtils-MakeMaker's test suite to fail in 02xs-dynamic.t, and all EUMM based XS packages generate a makefile with:

AR = ar

And afterwards, assuming you installed it regardless of test failure ( or even not running tests ), this eventually leads to breaking anything that consumes $Config{ar} expecting it to be the ar used to build "Perl itself" ( which is impossible, such a thing as ar didn't exist when Perl was built ).

However, there is a good workaround, as long as one invokes ./configure with:

-Dar=x86_64-pc-linux-gnu-ar  

(Or similar as appropriate for the target, just keep in mind this may not even be a gcc/binutils ar, but one provided by the llvm project)

Then it JustWorks:

So it seems this is just an inadequacy in Config not detecting AR appropriately, and also not bailing as appropriate when it would generate a bogus configuration.

Perl configuration

Summary of my perl5 (revision 5 version 30 subversion 2) configuration:

  Platform:
    osname=linux
    osvers=5.2.7-gentoo
    archname=x86_64-linux
    uname='linux localhost 5.2.7-gentoo #58 smp preempt fri aug 9 08:12:43 nzst 2019 x86_64 intel(r) core(tm) i5-2410m cpu @ 2.30ghz genuineintel gnulinux '
    config_args='-des -Dinstallprefix=/usr -Dinstallusrbinperl=n -Ui_xlocale -Di_ndbm -Di_gdbm -Di_db -DDEBUGGING=none -Dinc_version_list=5.30.1/x86_64-linux 5.30.1 5.30.0/x86_64-linux 5.30.0 5.28.0 5.26.2 5.26.1 5.24.2 5.24.1 5.24.0 5.22.2  -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dnoextensions=ODBM_File -Duseshrplib -Darchname=x86_64-linux -Dcc=x86_64-pc-linux-gnu-gcc -Dar=x86_64-pc-linux-gnu-ar -Doptimize=-O2 -pipe -mtune=native -march=native -fstack-protector-strong -fno-stack-protector -Dldflags=-Wl,-O1 -Wl,--as-needed -fstack-protector-strong -fno-stack-protector -Dprefix=/usr -Dsiteprefix=/usr/local -Dvendorprefix=/usr -Dscriptdir=/usr/bin -Dprivlib=/usr/lib64/perl5/5.30.2 -Darchlib=/usr/lib64/perl5/5.30.2/x86_64-linux -Dsitelib=/usr/local/lib64/perl5/5.30.2 -Dsitearch=/usr/local/lib64/perl5/5.30.2/x86_64-linux -Dvendorlib=/usr/lib64/perl5/vendor_perl/5.30.2 -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.30.2/x86_64-linux -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3 -Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.30.2 -Dlocincpth=/usr/include  -Dglibpth=/lib64 /usr/lib64  -Duselargefiles -Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost -Dperladmin=root@localhost -Ud_csh -Dsh=/bin/sh -Dtargetsh=/bin/sh -Uusenm -Ui_xlocale -Di_ndbm -Di_gdbm -Di_db -DDEBUGGING=none -Dinc_version_list=5.30.1/x86_64-linux 5.30.1 5.30.0/x86_64-linux 5.30.0 5.28.0 5.26.2 5.26.1 5.24.2 5.24.1 5.24.0 5.22.2  -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dnoextensions=ODBM_File'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='x86_64-pc-linux-gnu-gcc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -pipe -mtune=native -march=native -fstack-protector-strong -fno-stack-protector'
    cppflags='-fwrapv -fno-strict-aliasing -pipe'
    ccversion=''
    gccversion='9.3.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='x86_64-pc-linux-gnu-gcc'
    ldflags ='-Wl,-O1 -Wl,--as-needed -fstack-protector-strong -fno-stack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include-fixed /usr/lib
    libs=-lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-ldl -lm -lcrypt -lutil -lc
    libc=
    so=so
    useshrplib=true
    libperl=libperl.so.5.30.2
    gnulibc_version='2.31'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -pipe -mtune=native -march=native -fstack-protector-strong -fno-stack-protector -Wl,-O1 -Wl,--as-needed -fstack-protector-strong -fno-stack-protector'

Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
  Locally applied patches:
    gentoo/hints_hpux - Fix hpux hints
    gentoo/aix_soname - aix gcc detection and shared library soname support
    gentoo/EUMM-RUNPATH - https://bugs.gentoo.org/105054 cpan/ExtUtils-MakeMaker: drop $PORTAGE_TMPDIR from LD_RUN_PATH
    gentoo/config_over - Remove -rpath and append LDFLAGS to lddlflags
    gentoo/opensolaris_headers - Add headers for opensolaris
    gentoo/patchlevel - List packaged patches for perl-5.30.2-r1(#1) in patchlevel.h
    gentoo/cleanup-paths - Cleanup PATH and shrpenv
    gentoo/enc2xs - Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    gentoo/darwin-cc-ld - https://bugs.gentoo.org/297751 darwin: Use $CC to link
    gentoo/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    gentoo/interix - Fix interix hints
    gentoo/create_libperl_soname - https://bugs.gentoo.org/286840 Set libperl soname
    gentoo/mod_paths - Add /etc/perl to @INC
    gentoo/EUMM_perllocalpod - cpan/ExtUtils-MakeMaker: remove targets that generate perllocal.pod
    gentoo/drop_fstack_protector - https://bugs.gentoo.org/348557 Don't force -fstack-protector on everyone
    gentoo/usr_local - Configure: Don't include sources in /usr/local/ for compiling perl
    gentoo/D-SHA-CFLAGS - https://bugs.gentoo.org/506818 Do not set custom CFLAGS in cpan/Digest-SHA
    gentoo/io_socket_ip_tests - cpan/IO-Socket-IP: Disable network tests
    gentoo/tests - Fix EUMM podlocal tests
    gentoo/no-nsl-cl.patch -
    gentoo/no_porting_tests - Disable porting tests which create fun false-failures all over travis
    gentoo/pathtools_enoent - Disable PathTools tests which fails under sandboxing
    debian/cpan-missing-site-dirs - Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    debian/makemaker-pasthru - Pass LD settings through to subdirectories
    fixes/memoize_storable_nstore - [rt.cpan.org #77790] Memoize::Storable: respect 'nstore' option not respected
    fixes/podman-pipe - Better errors for man pages from standard input
    fixes/respect_umask - Respect umask during installation
    fixes/net_smtp_docs - [rt.cpan.org #36038] Document the Net::SMTP 'Port' option
    fixes/document_makemaker_ccflags - [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    fixes/parallel-manisort.patch - Fix parallel building
  Built under linux
  Compiled at May 18 2020 16:50:07
  @INC:
    /etc/perl
    /usr/local/lib64/perl5/5.30.2/x86_64-linux
    /usr/local/lib64/perl5/5.30.2
    /usr/lib64/perl5/vendor_perl/5.30.2/x86_64-linux
    /usr/lib64/perl5/vendor_perl/5.30.2
    /usr/local/lib64/perl5
    /usr/lib64/perl5/vendor_perl/5.30.1/x86_64-linux
    /usr/lib64/perl5/vendor_perl/5.30.1
    /usr/lib64/perl5/vendor_perl/5.30.0/x86_64-linux
    /usr/lib64/perl5/vendor_perl/5.30.0
    /usr/lib64/perl5/vendor_perl/5.28.0
    /usr/lib64/perl5/vendor_perl/5.26.2
    /usr/lib64/perl5/vendor_perl/5.26.1
    /usr/lib64/perl5/vendor_perl/5.24.2
    /usr/lib64/perl5/vendor_perl/5.24.1
    /usr/lib64/perl5/vendor_perl/5.24.0
    /usr/lib64/perl5/vendor_perl/5.22.2
    /usr/lib64/perl5/vendor_perl
    /usr/lib64/perl5/5.30.2/x86_64-linux
    /usr/lib64/perl5/5.30.2
Tux commented 4 years ago

On Sun, 17 May 2020 22:23:58 -0700, Kent Fredric notifications@github.com wrote:

Description We have developers on Gentoo testing some unusual setups, particularly:

  • There is no /usr/bin/gcc, /usr/bin/cc or /usr/bin/ar
  • There is no gcc, cc or ar in $PATH
  • AR= and CC= are both defined with correct values.

Under this configuration, perl ./configure fails to detect a correct ar, and later, embeds ar = ar in Config_heavy.pl

However, this failure does not stop perl from compiling, but it does make ExtUtils-MakeMaker's test suite to fail in 02xs-dynamic.t, and all EUMM based XS packages generate a makefile with:

AR = ar

And afterwards, assuming you installed it regardless of test failure ( or even not running tests ), this eventually leads to breaking anything that consumes $Config{ar} expecting it to be the ar used to build "Perl itself" ( which is impossible, such a thing as ar didn't exist when Perl was built ).

Your case seems clear, so the question that remains is: are cc and are the only two, or is it also others, like nm and ranlib?

If we have a defined set, would this mean that we can leave everything in place until the very last step where just before we set AR to 'ar' we should check if $AR is set to anything and use that instead. In sh talk

AR=ar
→
AR=${AR:-ar}

However, there is a good workaround, as long as one invokes ./configure with:

-Dar=x86_64-pc-linux-gnu-ar  

Interestingly, ar is set this way under crosscompiling

(Or similar as appropriate for the target, just keep in mind this may not even be a gcc/binutils ar, but one provided by the llvm project)

Then it JustWorks:

  • The aformentioned MakeMaker tests pass
  • No new test failures occur
  • Subsequent EUMM XS builds have a valid value for AR =
  • Large volumes of EUMM XS builds then work

So it seems this is just an inadequacy in Config not detecting AR appropriately, and also not bailing as appropriate when it would generate a bogus configuration.

Perl configuration

kentfredric commented 4 years ago

Your case seems clear, so the question that remains is: are cc and are the only two, or is it also others, like nm and ranlib?

There are others, but its not clear.

For instance, the same is happening with "ld", but I'm nowhere going to touch that one because too much perl code expects "ld" to be a ccld, not an actual "ld".

I'm sure eventually nm and ranlib may also be done this way, but our tester is starting out simple (for the most part, the end goal is to mimic non-gnu/linux platforms or cross-development tooling while still being run on a more popular architecture ).

I'm sure eventually I'll get bugs filed about code directly trying to invoke /usr/bin/ranlib , or something, just it hasn't happened yet.

kentfredric commented 4 years ago

Just got some more output from our tester.

The mention of gcc and cpp here are a little weird as we do pass -Dcc=, but perhaps some love is needed for G++ / CXX

kentfredric commented 4 years ago

As for LD, here's some background on why I'm not touching it with a barge pole:

Simply setting "LD=path/to/an/ld" breaks perl code, and has perl code authors telling you that you're wrong to set LD to an LD , ....

https://rt.cpan.org/Public/Bug/Display.html?id=120907 https://bugs.gentoo.org/641054

ld handling is a complete shit-show and I'm too afraid to touch it.

kentfredric commented 4 years ago

Further testing:

Building perl with -Dld=path/to/actual/ld breaks perl building:

cp DynaLoader.o ../../DynaLoader.o
make[1]: Leaving directory '/var/tmp/portage/dev-lang/perl-5.30.2-r1/work/perl-5.30.2/ext/DynaLoader'
rm -f libperl.so.5.30.2
x86_64-pc-linux-gnu-ld -o libperl.so.5.30.2 -shared -O2 -pipe -mtune=native -march=native -fstack-protector-strong -fno-stack-protector -Wl,-O1 -Wl,--as-needed -fstack-protector-strong -fno-stack-protector -Wl,-soname -Wl,libperl.so.5.30 op.o     perl.o  gv.o toke.o perly.o pad.o regcomp.o dump.o util.o mg.o reentr.o mro_core.o keywords.o hv.o av.o run.o pp_hot.o sv.o pp.o scope.o pp_ctl.o pp_sys.o doop.o doio.o regexec.o utf8.o taint.o deb.o universal.o globals.o perlio.o perlapi.o numeric.o mathoms.o locale.o pp_pack.o pp_sort.o caretx.o dquote.o time64.o   DynaLoader.o -ldl -lm -lcrypt -lutil -lc 
x86_64-pc-linux-gnu-ld: unrecognised emulation mode: arch=native
Supported emulations: aix5ppc aix5rs6 aixppc aixrs6 alpha alphavms arcv2elf arcv2elfx arcelf arclinux arclinux_nps arm_wince_pe armelf armelf_fbsd armelf_fuchsia armelf_linux armelf_linux_eabi armelf_linux_fdpiceabi armelf_nacl armelf_nbsd armelf_phoenix armelf_vxworks armelfb armelfb_fbsd armelfb_fuchsia armelfb_linux armelfb_linux_eabi armelfb_linux_fdpiceabi armelfb_nacl armelfb_nbsd armnto armpe armsymbian avr1 avr2 avr25 avr3 avr31 avr35 avr4 avr5 avr51 avr6 avrxmega1 avrxmega2 avrxmega3 avrxmega4 avrxmega5 avrxmega6 avrxmega7 avrtiny crisaout criself crislinux cskyelf cskyelf_linux d10velf d30v_e d30v_o d30velf elf32_dlx elf32_sparc elf32_sparc_sol2 elf32_sparc_vxworks elf32_spu elf32_tic6x_be elf32_tic6x_le elf32_tic6x_linux_be elf32_tic6x_linux_le elf32_tic6x_elf_be elf32_tic6x_elf_le elf32am33lin elf32bfin elf32bfinfd elf32cr16 elf32crx elf32epiphany elf32epiphany_4x4 elf32fr30 elf32frv elf32frvfd elf32ft32 elf32ip2k elf32iq10 elf32iq2000 elf32lm32 elf32lm32fd elf32lppc elf32lppclinux elf32lppcnto elf32lppcsim elf32m32c elf32mb_linux elf32mbel_linux elf32mcore elf32mep elf32metag elf32microblazeel elf32microblaze elf32moxie moxiebox elf32mt elf32or1k elf32or1k_linux elf32ppc elf32ppc_fbsd elf32ppclinux elf32ppcnto elf32ppcsim elf32ppcvxworks elf32ppcwindiss elf32lriscv elf32lriscv_ilp32f elf32lriscv_ilp32 elf32rl78 elf32rx elf32tilegx elf32tilegx_be elf32tilepro elf32vax elf32visium elf32xc16x elf32xc16xl elf32xc16xs elf32xstormy16 elf32xtensa elf32z80 elf_i386 elf_i386_be elf_i386_fbsd elf_i386_ldso elf_i386_nacl elf_i386_sol2 elf_i386_vxworks elf_iamcu elf_s390 h8300elf h8300elf_linux h8300helf h8300helf_linux h8300hnelf h8300self h8300self_linux h8300snelf h8300sxelf h8300sxelf_linux h8300sxnelf hppaelf hppalinux hppanbsd hppaobsd i386beos i386bsd i386go32 i386lynx i386moss i386msdos i386nto i386pe i386pe_posix m32relf m32relf_linux m32rlelf m32rlelf_linux m68hc11elf m68hc11elfb m68hc12elf m68hc12elfb m68kelf m68kelfnbsd m9s12zelf mcorepe mn10200 mn10300 msp430elf msp430X nds32elf nds32elf16m nds32elf_linux nds32belf nds32belf16m nds32belf_linux ns32knbsd nios2elf nios2linux pc532macha pdp11 pjelf pjlelf ppclynx ppcmacos ppcpe pruelf score3_elf score7_elf sh shelf shelf_fd shelf_linux shelf_nbsd shelf_nto shelf_uclinux shelf_vxworks shl shlelf shlelf_fd shlelf_linux shlelf_nbsd shlelf_nto shlelf_vxworks shpe tic30aout tic30coff tic3xcoff tic3xcoff_onchip tic4xcoff tic54xcoff v850 v850_rh850 vanilla vaxnbsd xgateelf z80 z8001 z8002 aarch64elf aarch64elf32 aarch64elfb aarch64elf32b aarch64cloudabi aarch64cloudabib aarch64fbsd aarch64fbsdb aarch64linux aarch64linuxb aarch64linux32 aarch64linux32b elf32_x86_64 elf32_x86_64_nacl elf32b4300 elf32bmip elf32bmipn32 elf32bsmip elf32btsmip elf32btsmip_fbsd elf32btsmipn32 elf32btsmipn32_fbsd elf32ebmip elf32ebmipvxworks elf32elmip elf32elmipvxworks elf32l4300 elf32lmip elf32lr5900 elf32lr5900n32 elf32lsmip elf32ltsmip elf32ltsmip_fbsd elf32ltsmipn32 elf32ltsmipn32_fbsd elf32mipswindiss elf64_aix elf64bpf elf64_ia64 elf64_ia64_fbsd elf64_ia64_vms elf64_s390 elf64_sparc elf64_sparc_fbsd elf64_sparc_sol2 elf64alpha elf64alpha_fbsd elf64alpha_nbsd elf64bmip elf64btsmip elf64btsmip_fbsd elf64hppa elf64lppc elf64lriscv elf64lriscv_lp64f elf64lriscv_lp64 elf64ltsmip elf64ltsmip_fbsd elf64mmix elf64ppc elf64ppc_fbsd elf64rdos elf64tilegx elf64tilegx_be elf_l1om elf_l1om_fbsd elf_k1om elf_k1om_fbsd elf_x86_64 elf_x86_64_cloudabi elf_x86_64_fbsd elf_x86_64_nacl elf_x86_64_sol2 hppa64linux i386pep mmo
make: *** [makefile:340: libperl.so.5.30.2] Error 1
make: *** Waiting for unfinished jobs....

This is despite this output:

Any additional ld flags (NOT including libraries)? [-Wl,-O1 -Wl,--as-needed -fstack-protector-strong -fno-stack-protector]

But this looks bad:

What command should be used to create dynamic libraries? [x86_64-pc-linux-gnu-ld]  
Any special flags to pass to x86_64-pc-linux-gnu-ld to create a dynamically loaded library? [-shared -O2 -pipe -mtune=native -march=native -fstack-protector-strong -fno-stack-protector]  
Any special flags to pass to x86_64-pc-linux-gnu-gcc to use dynamic linking? [-Wl,-E]  
ld supports scripting
kentfredric commented 4 years ago

https://metacpan.org/source/SHAY/perl-5.30.2/hints/linux.sh#L166-179

if [ -x /usr/bin/gcc ] ; then
    gcc=/usr/bin/gcc
else
    gcc=gcc
fi

case "$plibpth" in
'') plibpth=`LANG=C LC_ALL=C $gcc $ccflags $ldflags -print-search-dirs | grep libraries |
        cut -f2- -d= | tr ':' $trnl | grep -v 'gcc' | sed -e 's:/$::'`
    set X $plibpth # Collapse all entries on one line
    shift
    plibpth="$*"
    ;;
esac

Well that's annoying.