Enabling LTO (Was: p5.9.20210221.0: build fails)

[kloczek]

Here is the begin on my build process in which I'm regenearing ac/am/lt files and executing condfigure::

+ cd ctags-p5.9.20210221.0
+ autoreconf -fiv
autoreconf: export WARNINGS=
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: /usr/bin/autoconf --force
configure.ac:242: warning: The macro `AC_PROG_CC_C99' is obsolete.
configure.ac:242: You should run autoupdate.
./lib/autoconf/c.m4:1659: AC_PROG_CC_C99 is expanded from...
configure.ac:242: the top level
configure.ac:247: warning: The macro `AC_PROG_CC_C99' is obsolete.
configure.ac:247: You should run autoupdate.
./lib/autoconf/c.m4:1659: AC_PROG_CC_C99 is expanded from...
m4/ax_prog_cc_for_build.m4:38: AX_PROG_CC_FOR_BUILD is expanded from...
configure.ac:247: the top level
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:241: installing './compile'
configure.ac:35: installing './config.guess'
configure.ac:35: installing './config.sub'
configure.ac:28: installing './install-sh'
configure.ac:28: installing './missing'
Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/tags.am: warning: redefinition of 'ctags' ...
/usr/share/automake-1.16/am/program.am: ... 'ctags$(EXEEXT)' previously defined here
Makefile.am:38:   while processing program 'ctags'
autoreconf: './install-sh' is updated
autoreconf: './config.sub' is updated
autoreconf: './config.guess' is updated
autoreconf: Leaving directory '.'
+ CFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ CXXFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ FFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules'
+ FCFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules'
+ LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin'
+ CC=/usr/bin/gcc
+ CXX=/usr/bin/g++
+ FC=/usr/bin/gfortran
+ AR=/usr/bin/gcc-ar
+ NM=/usr/bin/gcc-nm
+ RANLIB=/usr/bin/gcc-ranlib
+ export CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CC CXX FC AR NM RANLIB
+ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --disable-silent-rules --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-etags --enable-json --enable-seccomp --enable-xml --enable-yaml
Universal Ctags, version 5.9.0
Linux 5.10.6-200.fc33.x86_64 #1 SMP Mon Jan 11 20:05:21 UTC 2021 x86_64
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '1000' is supported by ustar format... yes
checking whether GID '1000' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking build system type... x86_64-redhat-linux-gnu
checking host system type... x86_64-redhat-linux-gnu
checking for x86_64-redhat-linux-gnu-gcc... /usr/bin/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether /usr/bin/gcc accepts -g... yes
checking for /usr/bin/gcc option to enable C11 features... none needed
checking whether /usr/bin/gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of /usr/bin/gcc... none
checking for x86_64-redhat-linux-gnu-ranlib... /usr/bin/gcc-ranlib
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking whether byte ordering is bigendian... no
checking how to run the C preprocessor... /usr/bin/gcc -E
checking for x86_64-redhat-linux-gnu-gcc... no
checking for gcc... gcc
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... (cached) none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... none
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking how to run the C preprocessor... gcc -E
checking for cc for build... gcc
checking whether ln -s works... yes
checking for strip... strip
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for perl... no
checking for rst2man... /usr/bin/rst2man
checking whether rst2mn has --syntax-highlight option... yes
checking building in a git repository... no
checking for git... git
checking for timeout... timeout
checking for python3... python3
checking directory to use for temporary files... /tmp
checking for case-insensitive filenames... no
checking selected sort method... external sort utility
checking for sort... yes
checking if sort accepts our command line... yes
checking for direct.h... no
checking for dirent.h... yes
checking for fcntl.h... yes
checking for io.h... no
checking for stat.h... no
checking for types.h... no
checking for unistd.h... (cached) yes
checking for sys/dir.h... yes
checking for sys/stat.h... (cached) yes
checking for sys/types.h... (cached) yes
checking for sys/wait.h... yes
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking if L_tmpnam is defined in stdio.h... yes
checking for off_t... yes
checking for stdbool.h... yes
checking for an ANSI C-conforming const... yes
checking for typeof syntax and keyword spelling... typeof
checking if compiler supports statement expressions... yes
checking if struct stat contains st_ino... yes
checking for fnmatch... yes
checking for fnmatch.h... yes
checking for asprintf... yes
checking for strstr... yes
checking for strcasecmp... yes
checking for strncasecmp... yes
checking for mkstemp... yes
checking for opendir... yes
checking for strerror... yes
checking for truncate... yes
checking for setenv... yes
checking for regcomp... yes
checking if regcomp works... yes
checking for scandir... yes
checking for dirent.h... (cached) yes
checking for x86_64-redhat-linux-gnu-pkg-config... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBXML... yes
checking for JANSSON... yes
checking for SECCOMP... yes
checking for LIBYAML... yes
configure: checking for new missing prototypes
checking for iconv_open or libiconv_open with -liconv... no
checking for iconv_open... yes
checking for /usr/bin/gcc options needed to detect all undeclared functions... none needed
checking whether __environ is declared... yes
checking whether _NSGetEnviron is declared... no
checking whether strnlen is declared... yes
checking whether strnlen is declared for build... yes
checking for mblen... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating man/ctags.1.rst
config.status: creating man/ctags-client-tools.7.rst
config.status: creating man/ctags-incompatibilities.7.rst
config.status: creating man/ctags-optlib.7.rst
config.status: creating man/ctags-faq.7.rst
config.status: creating man/ctags-lang-julia.7.rst
config.status: creating man/ctags-lang-python.7.rst
config.status: creating man/ctags-lang-verilog.7.rst
config.status: creating man/ctags-lang-inko.7.rst
config.status: creating man/ctags-lang-r.7.rst
config.status: creating man/readtags.1.rst
config.status: creating config.h
config.status: executing depfiles commands

Than at the end final linking fails:

/usr/bin/gcc -DHAVE_CONFIG_H -I.  -I. -I. -I./main -I./libreadtags -DREADTAGS_DSL -I./dsl    -Wall -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -c -o dsl/readtags-es.o `test -f 'dsl/es.c' || echo './'`dsl/es.c
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
/usr/bin/rst2man --syntax-highlight=none man/ctags.1.rst man/ctags.1
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
gcc -g -O2  -DUSE_SYSTEM_STRNLEN -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin -o packcc ./misc/packcc/packcc.c
/usr/bin/ld: /tmp/ccXrfuA9.lto.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:5896: packcc] Error 1
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
make[1]: *** Waiting for unfinished jobs....
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
/usr/bin/gcc  -Wall -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none  -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin -o readtags libreadtags/readtags-readtags.o extra-cmds/readtags-printtags.o extra-cmds/readtags-readtags-cmd.o  dsl/readtags-es.o dsl/readtags-dsl.o dsl/readtags-qualifier.o dsl/readtags-sorter.o main/readtags-mio.o
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210221.0'
make: *** [Makefile:1359: all] Error 2

As you may see I'm passing to the configure env CC=/usr/bin/gcc however on final linking this $CC is not used because in configure.ac there are two compiler detectiom macros:

AC_PROG_CC                        << here
AC_PROG_CC_C99
AC_PROG_RANLIB
AC_C_BIGENDIAN

# https://www.gnu.org/software/autoconf-archive/ax_prog_cc_for_build.html
AX_PROG_CC_FOR_BUILD     << and here

[tkloczko@barrel ctags-p5.9.20210221.0]$ grep gcc Makefile
        main/dependency.h main/entry.h main/field.h main/gcc-attr.h \
CC = /usr/bin/gcc
CC_FOR_BUILD = gcc
CFLAGS = -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none
CPP = /usr/bin/gcc -E
CPP_FOR_BUILD = gcc -E
RANLIB = /usr/bin/gcc-ranlib
ac_ct_CC_FOR_BUILD = gcc
        main/gcc-attr.h         \

[tkloczko@barrel ctags-p5.9.20210221.0]$ grep gcc Makefile
        main/dependency.h main/entry.h main/field.h main/gcc-attr.h \
CC = /usr/bin/gcc                    << here
CC_FOR_BUILD = gcc               << here
CFLAGS = -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none
CPP = /usr/bin/gcc -E
CPP_FOR_BUILD = gcc -E
RANLIB = /usr/bin/gcc-ranlib
ac_ct_CC_FOR_BUILD = gcc                  << and here
        main/gcc-attr.h         \

Other small issies:

• AC_PROG_RANLIB is already marked as obsoleted and LT_INIT should be used
• instead using custom combination to build packetcc building that binaty should be put as noinst_PROGTRAM = packcc than it could be used and automake will guard to build packcc before it will be used and that binary will be not installed
• at the moment is a bit strange taggin convention used like p<version>.<date> despite fact that in configure c is not changeing line:

  AC_INIT([universal-ctags],[5.9.0])

  Is it any reason why it is used that way? :P Please consider use clear pattern for git tags which will match with what actially is used in AC_INIT

[kloczek]

Yet another small thing

[tkloczko@barrel ctags-p5.9.20210221.0]$ grep GNU configure.ac
#       of the GNU General Public License version 2 or (at your option) any
               [-D_GNU_SOURCE -D__USE_GNU])
REGCOMP_CPPFLAGS="-I${srcdir}/gnu_regex -D__USE_GNU"

Looks like -D__USE_GNU is added in two places. Maybe instead addinig it two two variables (REGCOMP_CPPFLAGS and EXTRA_CPPFLAGS) it would better to just add it to CFLAGS nd make configure.ac amd Makefile.am a bit simpler?

[kloczek]

And yet another small issue aroung CC detection.

[tkloczko@barrel ctags-p5.9.20210221.0]$ autoupdate
configure.ac:242: warning: AC_PROG_CC_C99 is obsolete; use AC_PROG_CC

[leleliu008]

@kloczek

Thank you for your reporting.

It seems that you are using GNU Autoconf latest version(2.70 or above), it changed much, and it reports warning: AC_PROG_CC_C99 is obsolete; use AC_PROG_CC, this warning is nothing to do with this issue, the real reason may be the CFLAGS and LDFLAGS you have set. Because I don't have redhat Linux, I have no choice to reproduce this issue but to guess.

try set CFLAGS_FOR_BUILD='' CPPFLAGS_FOR_BUILD='' LDFLAGS_FOR_BUILD=''

[kloczek]

t seems that you are using GNU Autoconf latest version(2.70 or above),

2.71 :) 2.70 had some seriouse bugs and 2.71 works in ~99% cases (except come really messy/old/not updated ac support).

it changed much, and it reports warning: AC_PROG_CC_C99 is obsolete; use AC_PROG_CC

That is obviouse but I've mentioned abot that as possible cleanup/simplification. Without that line still everything should be working on ac 2.69. This is why I've mentioned that in point after "Other small issies" :)

try set CFLAGS_FOR_BUILD='' CPPFLAGS_FOR_BUILD='' LDFLAGS_FOR_BUILD=''

More likely will try to fix that using RighWay(tm) probably this weekend :) I really don't like JFDIs :P

[masatake]

More likely will try to fix that using RighWay(tm) probably this weekend :)

I don't understand this English sentence. Who will try to fix that?

[kloczek]

me .. :P

[kloczek]

PR submitted :)

[k-takata]

So, as we discussed in #2893, we use AX_PROG_CC_FOR_BUILD for cross-compiling. Not only CC and CFLAGS but also CC_FOR_BUILD, CFLAGS_FOR_BUILD, and other *_FOR_BUILD flags should be set appropriately. Maybe this should be documented somewhere.

• AC_PROG_RANLIB is already marked as obsoleted and LT_INIT should be used

I'm not sure.

• instead using custom combination to build packetcc building that binaty should be put as noinst_PROGTRAM = packcc

Required for cross-compiling.

• at the moment is a bit strange taggin convention used like p<version>.<date> despite fact that in configure c is not changeing line:

The p style tags are tagged automatically and periodically, but we won't update AC_INIT automatically. When we create an official release, the tag will be in v6.0.0 style, and AC_INIT will be also updated.

Looks like -D__USE_GNU is added in two places.

Needs to be checked.

configure.ac:242: warning: AC_PROG_CC_C99 is obsolete; use AC_PROG_CC

We should just ignore this.

[masatake]

As summarized by @k-takata, only "appearing -D__USE_GNU twice" remains. However, I think is it not an issue.

[kloczek]

FYI autoconf 2.71 and build with use LTO and still ctags fails :/

+ cd ctags-p5.9.20210502.0
+ autoreconf -fiv
autoreconf: export WARNINGS=
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: /usr/bin/autoconf --force
configure.ac:251: warning: The macro `AC_PROG_CC_C99' is obsolete.
configure.ac:251: You should run autoupdate.
./lib/autoconf/c.m4:1659: AC_PROG_CC_C99 is expanded from...
configure.ac:251: the top level
configure.ac:256: warning: The macro `AC_PROG_CC_C99' is obsolete.
configure.ac:256: You should run autoupdate.
./lib/autoconf/c.m4:1659: AC_PROG_CC_C99 is expanded from...
m4/ax_prog_cc_for_build.m4:38: AX_PROG_CC_FOR_BUILD is expanded from...
configure.ac:256: the top level
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:250: installing './compile'
configure.ac:35: installing './config.guess'
configure.ac:35: installing './config.sub'
configure.ac:28: installing './install-sh'
configure.ac:28: installing './missing'
Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/tags.am: warning: redefinition of 'ctags' ...
/usr/share/automake-1.16/am/program.am: ... 'ctags$(EXEEXT)' previously defined here
Makefile.am:44:   while processing program 'ctags'
autoreconf: Leaving directory '.'
+ CFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ CXXFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ FFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules'
+ FCFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules'
+ LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin'
+ CC=/usr/bin/gcc
+ CXX=/usr/bin/g++
+ FC=/usr/bin/gfortran
+ AR=/usr/bin/gcc-ar
+ NM=/usr/bin/gcc-nm
+ RANLIB=/usr/bin/gcc-ranlib
+ export CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CC CXX FC AR NM RANLIB
+ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --disable-silent-rules --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --runstatedir=/run --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-iconv --enable-etags --enable-json --enable-seccomp --enable-xml --enable-yaml
Universal Ctags, version 5.9.0
Linux 5.11.0-155.fc35.x86_64 #1 SMP Mon Feb 15 21:33:29 UTC 2021 x86_64
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '1000' is supported by ustar format... yes
checking whether GID '1000' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking build system type... x86_64-redhat-linux-gnu
checking host system type... x86_64-redhat-linux-gnu
checking for x86_64-redhat-linux-gnu-gcc... /usr/bin/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether /usr/bin/gcc accepts -g... yes
checking for /usr/bin/gcc option to enable C11 features... none needed
checking whether /usr/bin/gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of /usr/bin/gcc... none
checking for x86_64-redhat-linux-gnu-ranlib... /usr/bin/gcc-ranlib
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking whether byte ordering is bigendian... no
checking how to run the C preprocessor... /usr/bin/gcc -E
checking for x86_64-redhat-linux-gnu-gcc... no
checking for gcc... gcc
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... (cached) none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... none
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking how to run the C preprocessor... gcc -E
checking for cc for build... gcc
checking whether ln -s works... yes
checking for strip... strip
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for perl... no
checking for rst2man... /usr/bin/rst2man
checking whether rst2mn has --syntax-highlight option... yes
checking building in a git repository... no
checking for git... git
checking for timeout... timeout
checking for python3... python3
checking directory to use for temporary files... /tmp
checking for case-insensitive filenames... no
checking selected sort method... external sort utility
checking for sort... yes
checking if sort accepts our command line... yes
checking for direct.h... no
checking for dirent.h... yes
checking for fcntl.h... yes
checking for io.h... no
checking for stat.h... no
checking for types.h... no
checking for unistd.h... (cached) yes
checking for sys/dir.h... yes
checking for sys/stat.h... (cached) yes
checking for sys/types.h... (cached) yes
checking for sys/wait.h... yes
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking if L_tmpnam is defined in stdio.h... yes
checking for off_t... yes
checking for stdbool.h... yes
checking for an ANSI C-conforming const... yes
checking for typeof syntax and keyword spelling... typeof
checking if compiler supports statement expressions... yes
checking if struct stat contains st_ino... yes
checking for fnmatch... yes
checking for fnmatch.h... yes
checking for asprintf... yes
checking for strstr... yes
checking for strcasecmp... yes
checking for strncasecmp... yes
checking for mkstemp... yes
checking for opendir... yes
checking for strerror... yes
checking for truncate... yes
checking for setenv... yes
checking for regcomp... yes
checking if regcomp works... yes
checking for scandir... yes
checking for dirent.h... (cached) yes
checking for x86_64-redhat-linux-gnu-pkg-config... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBXML... yes
checking for JANSSON... yes
checking for SECCOMP... yes
checking for LIBYAML... yes
configure: checking for new missing prototypes
checking for /usr/bin/gcc options needed to detect all undeclared functions... none needed
checking whether __environ is declared... yes
checking whether _NSGetEnviron is declared... no
checking whether strnlen is declared... yes
checking whether strnlen is declared for build... yes
checking for mblen... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating man/ctags.1.rst
config.status: creating man/ctags-client-tools.7.rst
config.status: creating man/ctags-incompatibilities.7.rst
config.status: creating man/ctags-optlib.7.rst
config.status: creating man/ctags-faq.7.rst
config.status: creating man/ctags-lang-julia.7.rst
config.status: creating man/ctags-lang-python.7.rst
config.status: creating man/ctags-lang-verilog.7.rst
config.status: creating man/ctags-lang-inko.7.rst
config.status: creating man/ctags-lang-r.7.rst
config.status: creating man/readtags.1.rst
config.status: creating config.h
config.status: executing depfiles commands
+ /usr/bin/make -O -j48 V=1 VERBOSE=1
echo > main/repoinfo.h

[..]

gcc -g -O2  -DPCC_USE_SYSTEM_STRNLEN -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin -o packcc ./misc/packcc/src/packcc.c
/usr/bin/ld: /tmp/ccw3ZKkB.lto.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:5907: packcc] Error 1
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210502.0'
make[1]: *** Waiting for unfinished jobs....
make[1]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210502.0'
/usr/bin/gcc  -Wall -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none  -Wl,-z,relro -Wl,--as-needed -Wl,--gc-sections -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto -flto-partition=none -fuse-linker-plugin -o readtags libreadtags/readtags-readtags.o extra-cmds/readtags-printtags.o extra-cmds/readtags-readtags-cmd.o  dsl/readtags-es.o dsl/readtags-dsl.o dsl/readtags-qualifier.o dsl/readtags-sorter.o main/readtags-mio.o
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ctags-p5.9.20210502.0'
make: *** [Makefile:1370: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.hjjKKJ (%build)

All because still is used second compiler detection :(

[hirooih]

I've just read this issue ticket.

So, as we discussed in #2893, we use AX_PROG_CC_FOR_BUILD for cross-compiling. Not only CC and CFLAGS but also CC_FOR_BUILD, CFLAGS_FOR_BUILD, and other *_FOR_BUILD flags should be set appropriately. Maybe this should be documented somewhere.

This is documented in docs/autotools.rst by @leleliu008. See #2747.

[masatake]

I cannot remember why I reopened this.

[kloczek]

Because despite changes which has been applied in mean time when LTO is used (without cross compile) build still fails.

[hirooih]

I see.

How about the followings?;

• For native compiling
  • CC_FOR_BUILD, CFLAGS_FOR_BUILD, and other *_FOR_BUILD flags copy CC, CFLAGS, and etc. respectively.
• For cross compiling (for a user who knows what he/she is doing)
  • Let the user specify CC, CC_FOR_BULID, CFLAGS, CFLAGS_FOR_BUILD and others.

Then we don't have to use AX_PROG_CC_FOR_BUILD.

---

I found this issue ticket during searching a clue for my problem of AX_PROG_CC_FOR_BUILD. I've been working to make use of the latest gnulib for regex engine. It works well now, if and only if I comment out AX_PROG_CC_FOR_BUILD. It does not work, if I just add the m4 directory of gnulib on the -I option of ACLOCAL_AMFLAGS.

[kloczek]

don't get me wrong but .. I really do not understand why onll ctags has special way to handle cross compilation if ac/am/lt handleas that mnatively. ctags is only package know to me which is handing those things that way.

[leleliu008]

@kloczek We would appreciate if you could give us a better solution, thanks.

[kloczek]

I have my own solution to those LTO problems and here are patches which I'm using now. ctags.zip

[hirooih]

I see.

I was wrong. I don't understand yet. LTO should not matter.

don't get me wrong but .. I really do not understand why onll ctags has special way to handle cross compilation if ac/am/lt handleas that mnatively. ctags is only package know to me which is handing those things that way.

packcc build by CC_FOR_BUILD is used to generate files to build ctags which use CC. Usually cross compilation need one compiler for a target machine. But we need two compiler.

[leleliu008]

@masatake we don't need to use AX_PROG_CC_FOR_BUILD now. because the packcc is written in C89 now. we don't need to check C99 now.

[hirooih]

@kloczek,

Could you try this?

./autogen.sh
./configure   # no option
make

If I understand correctly, make will fail.

---

@leleliu008,

we don't need to use AX_PROG_CC_FOR_BUILD now. because the packcc is written in C89 now. we don't need to check C99 now.

It seems that I don't understand yet... How values for CC_FOR_BUILD and etc. are determined without AX_PROG_CC_FOR_BUILD?

[masatake]

@hirooih, see #3051.

[kloczek]

@kloczek,

Could you try this?

./autogen.sh
./configure   # no option
make

It will work because that case is without LTO settings. If you want to replicate LTO issue just try below script:

CFLAGS="-flto=auto -flto-partition=none";
LDFLAGS="-flto=auto -flto-partition=none -fuse-linker-plugin";
CC="/usr/bin/gcc"; CXX="/usr/bin/g++"; FC="/usr/bin/gfortran";
AR="/usr/bin/gcc-ar"; NM="/usr/bin/gcc-nm"; RANLIB="/usr/bin/gcc-ranlib";
export CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CC CXX FC AR NM RANLIB;
./configure
make

[masatake]

CFLAGS="-flto=auto -flto-partition=none"; LDFLAGS="-flto=auto -flto-partition=none -fuse-linker-plugin"; CC="/usr/bin/gcc"; CXX="/usr/bin/g++"; FC="/usr/bin/gfortran"; AR="/usr/bin/gcc-ar"; NM="/usr/bin/gcc-nm"; RANLIB="/usr/bin/gcc-ranlib"; export CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CC CXX FC AR NM RANLIB; ./configure make

With the change proposed in https://github.com/universal-ctags/ctags/pull/3051, ctags is built successfully with the LTO settings.

[hirooih]

@kloczek,

If you want to replicate LTO issue just try below script:

CFLAGS="-flto=auto -flto-partition=none";
LDFLAGS="-flto=auto -flto-partition=none -fuse-linker-plugin";
CC="/usr/bin/gcc"; CXX="/usr/bin/g++"; FC="/usr/bin/gfortran";
AR="/usr/bin/gcc-ar"; NM="/usr/bin/gcc-nm"; RANLIB="/usr/bin/gcc-ranlib";
export CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CC CXX FC AR NM RANLIB;
./configure
make

It works on my environment without applying the fix of #3051. I've confirmed for sure.

What you have to do is;

CFLAGS="-flto=auto -flto-partition=none";
LDFLAGS="-flto=auto -flto-partition=none -fuse-linker-plugin";
CC="/usr/bin/gcc"; CXX="/usr/bin/g++"; FC="/usr/bin/gfortran";
AR="/usr/bin/gcc-ar"; NM="/usr/bin/gcc-nm"; RANLIB="/usr/bin/gcc-ranlib";
CC_FOR_BUILD=$CC CFLAGS_FOR_BUILD=$CFLAGS  LDFLAGS_FOR_BUILD=$LDFLAGS;
export CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS CC CXX FC AR NM RANLIB CC_FOR_BUILD CFLAGS_FOR_BUILD LDFLAGS_FOR_BUILD;
./configure
make

or a little bit simpler;

CFLAGS="-flto=auto -flto-partition=none" \
LDFLAGS="-flto=auto -flto-partition=none -fuse-linker-plugin" \
CC=/usr/bin/gcc CXX=/usr/bin/g++ FC=/usr/bin/gfortran \
AR=/usr/bin/gcc-ar NM=/usr/bin/gcc-nm RANLIB=/usr/bin/gcc-ranlib \
CC_FOR_BUILD=$CC CFLAGS_FOR_BUILD="$CFLAGS"  LDFLAGS_FOR_BUILD="$LDFLAGS" \ 
./configure
make

as @k-takata wrote;

So, as we discussed in #2893, we use AX_PROG_CC_FOR_BUILD for cross-compiling. Not only CC and CFLAGS but also CC_FOR_BUILD, CFLAGS_FOR_BUILD, and other *_FOR_BUILD flags should be set appropriately. Maybe this should be documented somewhere.

Again could you try the following without applying without applying the fix of #3051?

./autogen.sh
./configure   # no option
make

If it works, show me how packcc is compiled.

---

@hirooih, see #3051.

Now I understand. The configure script now works as;

• For native compiling
  • use $CC, $CFLAGS, etc.
• For cross compiling
  • For packcc; use $CFLAGS_FOR_BUILD if defined, otherwise cc
  • For others: use $CC, $CFLAGS, etc.

[hirooih]

Sorry.

Again could you try the following without applying without applying the fix of #3051?

Should be;

Again could you try the following without applying the fix of #3051?

But it should fail even with the fix of #3051, if I understand correctly.

If it works, show me how packcc is compiled.

[masatake]

I'm optimizing ruby.c. After working for a while, I found that I was doing LTO manually. I want to enable LTO in our Makefile(.am) instead of enabling it in the process of building a rpm package for ctags.

Do you know how I can do it?

I read https://stackoverflow.com/questions/27241011/using-flto-with-autotools

The procedure on the page requires users to specify -flto manually. I don't want to add such a step to our document.

Without -lfto:

[yamato@dev64]~/var/codebase% for x in Ruby ; do ./codebase ctags --ctags ~/var/ctags-github/ctags $x; done
version: 28a0487d
features: +wildcards +regex +iconv +option-directory +xpath +json +interactive +sandbox +yaml +packcc +optscript +pcre2
log: results/28a0487d,Ruby................,..........,time......,default...,2024-01-13-07:14:42.log
tagsoutput: /dev/null
cmdline: + /home/yamato/var/ctags-github/ctags --quiet --options=NONE --sort=no --options=profile.d/maps --totals=yes --languages=Ruby -o - -R code/katello code/metasploit-framework code/puppet code/rails code/redmine code/rspec-core code/ruby code/theforeman-puppet code/vagrant
20744 files, 2852961 lines (88272 kB) scanned in 4.2 seconds (21062 kB/s)
185632 tags added to tag file

real    0m4.384s
user    0m4.059s
sys 0m0.299s
+ set +x

With -lfto:

[yamato@dev64]~/var/codebase% for x in Ruby ; do ./codebase ctags --ctags ~/var/ctags-github/ctags $x; done
version: 28a0487d
features: +wildcards +regex +iconv +option-directory +xpath +json +interactive +sandbox +yaml +packcc +optscript +pcre2
log: results/28a0487d,Ruby................,..........,time......,default...,2024-01-13-07:15:42.log
tagsoutput: /dev/null
cmdline: + /home/yamato/var/ctags-github/ctags --quiet --options=NONE --sort=no --options=profile.d/maps --totals=yes --languages=Ruby -o - -R code/katello code/metasploit-framework code/puppet code/rails code/redmine code/rspec-core code/ruby code/theforeman-puppet code/vagrant
20744 files, 2852961 lines (88272 kB) scanned in 3.3 seconds (26730 kB/s)
185632 tags added to tag file

real    0m3.487s
user    0m3.180s
sys 0m0.288s
+ set +x

[kloczek]

It is well known that libtool is BROKEN from many years by putting linking options in wrong order on liking. Result of use liker options DEPENDS on order of options. They are working only for everything added AFTER exact option. If you will put LTO options at the end of all params it will be NO LTO applied applied on linking. That what happens with unpatched libtool. Result of those libtool bugs is that -Wl,--as-needed is not working. This why Fedora has so many hacks spread across so many spec files to fis use of that option. However if correct fix will be applied directly in libtool not only -Wl,--as-needed will be placed in correct place but LTO liker options as well. Fedora and all other distros are fixes all those issues spreading them across almost all packages build procedures. In reality all what is needed is apply such fix in ONE PLACE .. libtool.

Libtool issues have been reported MANY times. Even some partial patches have been posted many times (it needs to be fixed ltmain.sh). First you need to have fixed libtool package than before start configure all GNU autotools needs to be regenerated (easiest way to do that is to execute autoreconf -fiv). Example from distro on which I'm working

[tkloczko@pers-jacek SPECS]$ grep -l "^autoreconf -fiv" *spec | wc -l
1004

Prepare above was not easy because many projects are not ready for example to be used with latest autoconf (I'm using autoconf 2.72)

[tkloczko@pers-jacek SPECS]$ grep "^Patch:.*%{name}-ac_fixes.patch" -l *spec | wc -l
46

In other words ~5% all projects which are using GBU autotools are not ready for latest autoconf.

ctags for example wrongly uses obsoleted with latest autoconf AC_PROG_CC_C99 aclocal macro. I've reported that and even proposed necessary patch. You can find it here https://github.com/universal-ctags/ctags/pull/2893.

[masatake]

@kloczek Thank you for your input. However, I need help understanding the points. ctags doesn't use libtool. So, we must just put -flto to the command lines for compiling and linking if the compiler supports the option. Am I wrong?