Build succeeds with flex 2.6.3, but fails with flex 2.6.4

[lyu]

Describe the bug If using flex 2.6.4, build fails with segmentation faults in m4, while everything works if I use flex 2.6.3 instead.

This issue is also mentioned here.

Expected behavior Build with flex 2.6.4 works.

To Reproduce flex is configured as: ./configure --prefix=$PREFIX --disable-shared --disable-nls

doxygen is configured as: cmake -DCMAKE_BUILD_TYPE=Release -Dbuild_doc=OFF -Dbuild_wizard=OFF

Version I've tried doxygen 1.8.18, 1.9.1, and the latest master 97415f7

CMake Log

Click to expand

``` -- The C compiler identification is GNU 10.2.0 -- The CXX compiler identification is GNU 10.2.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /lustre/home/user/.local/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /lustre/home/user/.local/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found PythonInterp: /lustre/home/user/.local/bin/python3 (found version "3.9.2") -- Found FLEX: /lustre/home/user/.local/bin/flex (found version "2.6.4") -- Found BISON: /lustre/home/user/.local/bin/bison (found version "3.7.6") -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Looking for iconv_open -- Looking for iconv_open - found -- Performing Test ICONV_COMPILES -- Performing Test ICONV_COMPILES - Success -- Found Iconv: In glibc -- One (and only one) of the ICONV_ACCEPTS_... tests must pass -- Performing Test ICONV_ACCEPTS_NONCONST_INPUT -- Performing Test ICONV_ACCEPTS_NONCONST_INPUT - Success -- Performing Test ICONV_ACCEPTS_CONST_INPUT -- Performing Test ICONV_ACCEPTS_CONST_INPUT - Failed -- The javacc executable not found, using existing files -- Configuring done -- Generating done -- Build files have been written to: /tmp/doxygen_objdir make: Entering directory '/tmp/doxygen_objdir' make[1]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target md5 Scanning dependencies of target lodepng make[2]: Entering directory '/tmp/doxygen_objdir' [ 1%] Generating ../generated_src/mscgen_lexer.l.h Scanning dependencies of target check_git_repository [ 2%] [BISON][mscgen_language] Building parser with bison 3.7.6 Scanning dependencies of target check_doxygen_version make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Leaving directory '/tmp/doxygen_objdir' [ 2%] [FLEX][mscgen_lexer] Building scanner with flex 2.6.4 [ 3%] Generating ../generated_src/xml.l.h make[2]: Leaving directory '/tmp/doxygen_objdir' [ 3%] [FLEX][xml] Building scanner with flex 2.6.4 make[2]: Leaving directory '/tmp/doxygen_objdir' Scanning dependencies of target generate_configvalues_header make[2]: Leaving directory '/tmp/doxygen_objdir' [ 4%] Generating /tmp/doxygen_objdir/generated_src/lang_cfg.h [ 5%] [FLEX][configimpl] Building scanner with flex 2.6.4 make[2]: Entering directory '/tmp/doxygen_objdir' [ 5%] Generating ../generated_src/configimpl.l.h [ 6%] Generating ../generated_src/configoptions.cpp make[2]: Entering directory '/tmp/doxygen_objdir' [ 6%] Generating ../generated_src/configvalues.cpp make[2]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 6%] Generating ../generated_src/configvalues.h make[2]: Entering directory '/tmp/doxygen_objdir' [ 7%] Generating ../generated_src/xmlcode.l.h [ 8%] [BISON][constexp] Building parser with bison 3.7.6 [ 9%] Generating ../generated_src/ce_parse.h [ 10%] [FLEX][commentcnv] Building scanner with flex 2.6.4 [ 10%] [FLEX][code] Building scanner with flex 2.6.4 [ 10%] Generating ../generated_src/code.l.h [ 10%] [FLEX][commentscan] Building scanner with flex 2.6.4 [ 10%] Generating ../generated_src/commentscan.l.h [ 10%] Checking the git repository for changes... [ 11%] Building C object libmd5/CMakeFiles/md5.dir/md5.c.o [ 12%] Generating ../generated_src/commentcnv.l.h [ 13%] [FLEX][constexp] Building scanner with flex 2.6.4 [ 13%] Generating ../generated_src/configvalues.h [ 13%] Checking the doxygen version for changes... [ 13%] Generating ../generated_src/constexp.l.h [ 13%] [FLEX][declinfo] Building scanner with flex 2.6.4 [ 14%] Generating ../generated_src/declinfo.l.h /tmp/doxygen_src/src/constexp.y:35.1-25: warning: deprecated directive: ‘%name-prefix "constexpYY"’, use ‘%define api.prefix {constexpYY}’ [-Wdeprecated] /tmp/doxygen_src/src/constexp.y:35.1-25 35 | : %nwarning:am e-deprecated directive: ‘%name-prefix "constexpYY"’, use ‘%define api.prefix {constexpYY}’pr [efi-Wdeprecatedx] "constexpYY" [ 15%] Building CXX object liblodepng/CMakeFiles/lodepng.dir/lodepng.cpp.o | ^~~~~ 35 | ~%~n~a~m~e~-~p~r~e~f~i~x~ ~"~c~o~n~s~t~e x | p%define api.prefix {constexpYY}Y Y" | ^~~~~~~~~~~~~~~~~~~~~~~~~ | %define api.prefix {constexpYY} [ 16%] Generating ../generated_src/defargs.l.h [ 16%] [FLEX][defargs] Building scanner with flex 2.6.4 [ 16%] [FLEX][doctokenizer] Building scanner with flex 2.6.4 make[2]: Leaving directory '/tmp/doxygen_objdir' [ 16%] Generating ../generated_src/doctokenizer.l.h [ 17%] Generating ../generated_src/fortrancode.l.h [ 17%] [FLEX][fortranscanner] Building scanner with flex 2.6.4 [ 18%] [FLEX][fortrancode] Building scanner with flex 2.6.4 [ 18%] Built target check_doxygen_version [ 19%] Generating ../generated_src/fortranscanner.l.h [ 19%] [FLEX][lexcode] Building scanner with flex 2.6.4 [ 20%] Generating ../generated_src/lexcode.l.h [ 20%] [FLEX][lexscanner] Building scanner with flex 2.6.4 [ 20%] Generating ../generated_src/lexscanner.l.h [ 21%] [FLEX][pre] Building scanner with flex 2.6.4 [ 22%] Generating ../generated_src/pre.l.h [ 22%] [FLEX][pycode] Building scanner with flex 2.6.4 [ 22%] Generating ../generated_src/pycode.l.h [ 23%] [FLEX][pyscanner] Building scanner with flex 2.6.4 [ 24%] Generating ../generated_src/pyscanner.l.h [ 24%] Generating /tmp/doxygen_objdir/generated_src/resources.cpp [ 24%] [FLEX][scanner] Building scanner with flex 2.6.4 Scanning dependencies of target xml [ 24%] Generating ../generated_src/scanner.l.h make[2]: Leaving directory '/tmp/doxygen_objdir' [ 25%] Generating ../generated_src/sqlcode.l.h [ 26%] [FLEX][sqlcode] Building scanner with flex 2.6.4 make[2]: Entering directory '/tmp/doxygen_objdir' [ 26%] Generating ../generated_src/vhdlcode.l.h [ 26%] [FLEX][vhdlcode] Building scanner with flex 2.6.4 [ 26%] [FLEX][xmlcode] Building scanner with flex 2.6.4 [ 26%] Building CXX object libxml/CMakeFiles/xml.dir/__/generated_src/xml.cpp.o make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Leaving directory '/tmp/doxygen_objdir' [ 26%] Built target generate_configvalues_header make[2]: Entering directory '/tmp/doxygen_objdir' [ 26%] Built target check_git_repository [ 26%] Generating ../generated_src/VhdlParser_adj.cc make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target doxygen_version /tmp/doxygen_src/src/constexp.y: warning: fix-its can be applied. Rerun with option '--update'. [-Wother] make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 26%] Building CXX object libversion/CMakeFiles/doxygen_version.dir/__/generated_src/doxyversion.cpp.o [ 27%] Building CXX object libversion/CMakeFiles/doxygen_version.dir/__/generated_src/gitversion.cpp.o [ 27%] Building CXX object libversion/CMakeFiles/doxygen_version.dir/fullversion.cpp.o /tmp/doxygen_src/src/constexp.y: warning: fix-its can be applied. Rerun with option '--update'. [-Wother] Scanning dependencies of target vhdlparser make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 29%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/TokenMgrError.cc.o [ 29%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/ParseException.cc.o [ 29%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/CharStream.cc.o [ 29%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/Token.cc.o [ 30%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/__/generated_src/VhdlParser_adj.cc.o [ 30%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/VhdlParserTokenManager.cc.o [ 31%] Linking CXX static library ../lib/libdoxygen_version.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 31%] Built target doxygen_version Scanning dependencies of target mscgen make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 33%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd_security.c.o [ 33%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd.c.o [ 33%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdfontt.c.o [ 33%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdtables.c.o [ 33%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd_color.c.o [ 33%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdfonts.c.o [ 34%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdhelpers.c.o [ 34%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd_lodepng.c.o [ 35%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_adraw.c.o [ 35%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_gd_out.c.o [ 36%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_ps_out.c.o [ 36%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_null_out.c.o [ 37%] Building CXX object libmscgen/CMakeFiles/mscgen.dir/__/generated_src/mscgen_language.cpp.o [ 37%] Building CXX object libmscgen/CMakeFiles/mscgen.dir/__/generated_src/mscgen_lexer.cpp.o Scanning dependencies of target doxycfg [ 37%] Linking C static library ../lib/libmd5.a [ 38%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_api.c.o make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 39%] Building CXX object src/CMakeFiles/doxycfg.dir/__/generated_src/configimpl.cpp.o make[2]: Leaving directory '/tmp/doxygen_objdir' [ 39%] Built target md5 [ 39%] Building CXX object src/CMakeFiles/doxycfg.dir/__/generated_src/configoptions.cpp.o [ 40%] Building CXX object src/CMakeFiles/doxycfg.dir/__/generated_src/configvalues.cpp.o [ 40%] Building CXX object src/CMakeFiles/doxycfg.dir/portable.cpp.o [ 40%] Building CXX object src/CMakeFiles/doxycfg.dir/message.cpp.o [ 41%] Building C object src/CMakeFiles/doxycfg.dir/portable_c.c.o [ 41%] Building CXX object src/CMakeFiles/doxycfg.dir/debug.cpp.o [ 41%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_msc.c.o [ 42%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_safe.c.o [ 42%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_svg_out.c.o [ 43%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_usage.c.o [ 43%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_utf8.c.o make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:175: generated_src/code.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/code.cpp' make[2]: *** Waiting for unfinished jobs.... make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:203: generated_src/fortranscanner.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/fortranscanner.cpp' /lustre/home/user/.local/bin/m4:stdin:679: ERROR: end of file in string make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:195: generated_src/doctokenizer.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/doctokenizer.cpp' /lustre/home/user/.local/bin/m4:make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:227: generated_src/scanner.cpp] Segmentation fault (core dumped) stdin:728: ERROR: end of file in string make[2]: *** Deleting file 'generated_src/scanner.cpp' make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:215: generated_src/pre.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/pre.cpp' [ 43%] Linking CXX static library ../lib/liblodepng.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 43%] Built target lodepng make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:199: generated_src/fortrancode.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/fortrancode.cpp' [ 44%] Linking CXX static library ../lib/libxml.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 44%] Built target xml [ 44%] Linking CXX static library ../lib/libmscgen.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 44%] Built target mscgen make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:235: generated_src/vhdlcode.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/vhdlcode.cpp' make[2]: Leaving directory '/tmp/doxygen_objdir' make[1]: *** [CMakeFiles/Makefile2:540: src/CMakeFiles/doxymain.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... [ 45%] Linking CXX static library ../lib/libdoxycfg.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 45%] Built target doxycfg [ 46%] Linking CXX static library ../lib/libvhdlparser.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 46%] Built target vhdlparser make[1]: Leaving directory '/tmp/doxygen_objdir' make: *** [Makefile:182: all] Error 2 make: Leaving directory '/tmp/doxygen_objdir' ```

[albert-github]

A bit strange. The build log is a bit hard to read as it looks like it is a parallel build (normally good, but for debugging sometimes a bit problematic) and also it looks like that the make runs further even though there are some serious problems.

I see you use

• flex 2.6.4 i.e. the latest version
• bison 3.7.6 i.e. the latest version

I use the latest default versions for Cygwin:

• flex 2.6.4
• bison 3.0.4 (but this version I don't see at https://github.com/akimd/bison, so I'm not sure whether or not they use the same source tree though it looks like they do seen the message in the given repository).

Some initial questions:

• did you build both yourself and with which options?
• did you build flex with --disable-nls why?

Can you try:

• building without the --disable-nls and see what happens
• looks like you build in source why why not using:

  mkdir build
  cd build
  cmake -DCMAKE_BUILD_TYPE=Release -Dbuild_doc=OFF -Dbuild_wizard=OFF ..
  make

• show the exact build commands you used
• do a non-parallel build

[lyu]

I am working on a university-owned cluster that runs CentOS 8. I do software development and the system packages are too old so I usually install a lot of stuff under my $HOME/.local, including all the usual build tools, this is the background.

I built flex with --disable-nls because I don't need NLS. But following your suggestion, now I install it with ./configure --prefix=$PREFIX --disable-shared (the --disable-shared is required due to some other issue in flex).

As requested, here are the exact build commands I used to do a serial out-of-tree build of doxygen:

PREFIX=$HOME/.local
export CC=$PREFIX/bin/gcc
export CXX=$PREFIX/bin/g++

git clone --depth 1 https://github.com/doxygen/doxygen doxygen_src
cmake -S doxygen_src                            \
      -B doxygen_objdir                         \
      -DCMAKE_BUILD_TYPE=Release                \
      -Dbuild_doc=OFF                           \
      -Dbuild_wizard=OFF                        \
      -DPYTHON_EXECUTABLE="$PREFIX/bin/python3" \
      -DCMAKE_INSTALL_PREFIX="$PREFIX"          \
      -DCMAKE_C_COMPILER="$CC"                  \
      -DCMAKE_CXX_COMPILER="$CXX"
make -C doxygen_objdir

Again, with commit 97415f7, the build log is:

Click to expand

``` -- The C compiler identification is GNU 10.2.0 -- The CXX compiler identification is GNU 10.2.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /lustre/home/user/.local/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /lustre/home/user/.local/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found PythonInterp: /lustre/home/user/.local/bin/python3 (found version "3.9.2") -- Found FLEX: /lustre/home/user/.local/bin/flex (found version "2.6.4") -- Found BISON: /lustre/home/user/.local/bin/bison (found version "3.7.6") -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Looking for iconv_open -- Looking for iconv_open - found -- Performing Test ICONV_COMPILES -- Performing Test ICONV_COMPILES - Success -- Found Iconv: In glibc -- One (and only one) of the ICONV_ACCEPTS_... tests must pass -- Performing Test ICONV_ACCEPTS_NONCONST_INPUT -- Performing Test ICONV_ACCEPTS_NONCONST_INPUT - Success -- Performing Test ICONV_ACCEPTS_CONST_INPUT -- Performing Test ICONV_ACCEPTS_CONST_INPUT - Failed -- The javacc executable not found, using existing files -- Configuring done -- Generating done -- Build files have been written to: /tmp/doxygen_objdir make: Entering directory '/tmp/doxygen_objdir' make[1]: Entering directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target md5 make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 1%] Building C object libmd5/CMakeFiles/md5.dir/md5.c.o [ 1%] Linking C static library ../lib/libmd5.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 1%] Built target md5 make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target lodepng make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 2%] Building CXX object liblodepng/CMakeFiles/lodepng.dir/lodepng.cpp.o [ 2%] Linking CXX static library ../lib/liblodepng.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 2%] Built target lodepng make[2]: Entering directory '/tmp/doxygen_objdir' [ 3%] Generating ../generated_src/mscgen_lexer.l.h [ 4%] [BISON][mscgen_language] Building parser with bison 3.7.6 [ 4%] [FLEX][mscgen_lexer] Building scanner with flex 2.6.4 Scanning dependencies of target mscgen make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 4%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd.c.o [ 5%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd_security.c.o [ 5%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdfontt.c.o [ 6%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdtables.c.o [ 6%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd_color.c.o [ 6%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdfonts.c.o [ 7%] Building C object libmscgen/CMakeFiles/mscgen.dir/gdhelpers.c.o [ 7%] Building C object libmscgen/CMakeFiles/mscgen.dir/gd_lodepng.c.o [ 8%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_adraw.c.o [ 8%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_gd_out.c.o [ 9%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_ps_out.c.o [ 9%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_null_out.c.o [ 10%] Building CXX object libmscgen/CMakeFiles/mscgen.dir/__/generated_src/mscgen_language.cpp.o [ 10%] Building CXX object libmscgen/CMakeFiles/mscgen.dir/__/generated_src/mscgen_lexer.cpp.o [ 11%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_api.c.o [ 11%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_msc.c.o [ 12%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_safe.c.o [ 12%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_svg_out.c.o [ 13%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_usage.c.o [ 13%] Building C object libmscgen/CMakeFiles/mscgen.dir/mscgen_utf8.c.o [ 13%] Linking CXX static library ../lib/libmscgen.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 13%] Built target mscgen make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target check_git_repository make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 13%] Checking the git repository for changes... make[2]: Leaving directory '/tmp/doxygen_objdir' [ 13%] Built target check_git_repository make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target check_doxygen_version make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 13%] Checking the doxygen version for changes... make[2]: Leaving directory '/tmp/doxygen_objdir' [ 13%] Built target check_doxygen_version make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target doxygen_version make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 13%] Building CXX object libversion/CMakeFiles/doxygen_version.dir/__/generated_src/doxyversion.cpp.o [ 14%] Building CXX object libversion/CMakeFiles/doxygen_version.dir/__/generated_src/gitversion.cpp.o [ 14%] Building CXX object libversion/CMakeFiles/doxygen_version.dir/fullversion.cpp.o [ 15%] Linking CXX static library ../lib/libdoxygen_version.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 15%] Built target doxygen_version make[2]: Entering directory '/tmp/doxygen_objdir' [ 16%] Generating ../generated_src/xml.l.h [ 16%] [FLEX][xml] Building scanner with flex 2.6.4 Scanning dependencies of target xml make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 16%] Building CXX object libxml/CMakeFiles/xml.dir/__/generated_src/xml.cpp.o [ 17%] Linking CXX static library ../lib/libxml.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 17%] Built target xml make[2]: Entering directory '/tmp/doxygen_objdir' Scanning dependencies of target generate_configvalues_header make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 17%] Generating ../generated_src/configvalues.h make[2]: Leaving directory '/tmp/doxygen_objdir' [ 17%] Built target generate_configvalues_header make[2]: Entering directory '/tmp/doxygen_objdir' [ 17%] Generating ../generated_src/VhdlParser_adj.cc Scanning dependencies of target vhdlparser make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 18%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/CharStream.cc.o [ 18%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/ParseException.cc.o [ 19%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/Token.cc.o [ 19%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/TokenMgrError.cc.o [ 20%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/__/generated_src/VhdlParser_adj.cc.o [ 20%] Building CXX object vhdlparser/CMakeFiles/vhdlparser.dir/VhdlParserTokenManager.cc.o [ 21%] Linking CXX static library ../lib/libvhdlparser.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 21%] Built target vhdlparser make[2]: Entering directory '/tmp/doxygen_objdir' [ 22%] Generating /tmp/doxygen_objdir/generated_src/lang_cfg.h [ 23%] [FLEX][configimpl] Building scanner with flex 2.6.4 [ 23%] Generating ../generated_src/configimpl.l.h [ 24%] Generating ../generated_src/configoptions.cpp [ 24%] Generating ../generated_src/configvalues.cpp Scanning dependencies of target doxycfg make[2]: Leaving directory '/tmp/doxygen_objdir' make[2]: Entering directory '/tmp/doxygen_objdir' [ 25%] Building CXX object src/CMakeFiles/doxycfg.dir/__/generated_src/configimpl.cpp.o [ 25%] Building CXX object src/CMakeFiles/doxycfg.dir/__/generated_src/configoptions.cpp.o [ 26%] Building CXX object src/CMakeFiles/doxycfg.dir/__/generated_src/configvalues.cpp.o [ 26%] Building CXX object src/CMakeFiles/doxycfg.dir/portable.cpp.o [ 27%] Building C object src/CMakeFiles/doxycfg.dir/portable_c.c.o [ 27%] Building CXX object src/CMakeFiles/doxycfg.dir/message.cpp.o [ 27%] Building CXX object src/CMakeFiles/doxycfg.dir/debug.cpp.o [ 28%] Linking CXX static library ../lib/libdoxycfg.a make[2]: Leaving directory '/tmp/doxygen_objdir' [ 28%] Built target doxycfg make[2]: Entering directory '/tmp/doxygen_objdir' [ 29%] Generating ../generated_src/xmlcode.l.h [ 30%] [BISON][constexp] Building parser with bison 3.7.6 /tmp/doxygen_src/src/constexp.y:35.1-25: warning: deprecated directive: ‘%name-prefix "constexpYY"’, use ‘%define api.prefix {constexpYY}’ [-Wdeprecated] 35 | %name-prefix "constexpYY" | ^~~~~~~~~~~~~~~~~~~~~~~~~ | %define api.prefix {constexpYY} /tmp/doxygen_src/src/constexp.y: warning: fix-its can be applied. Rerun with option '--update'. [-Wother] [ 31%] Generating ../generated_src/ce_parse.h /tmp/doxygen_src/src/constexp.y:35.1-25: warning: deprecated directive: ‘%name-prefix "constexpYY"’, use ‘%define api.prefix {constexpYY}’ [-Wdeprecated] 35 | %name-prefix "constexpYY" | ^~~~~~~~~~~~~~~~~~~~~~~~~ | %define api.prefix {constexpYY} /tmp/doxygen_src/src/constexp.y: warning: fix-its can be applied. Rerun with option '--update'. [-Wother] [ 31%] [FLEX][code] Building scanner with flex 2.6.4 make[2]: *** [src/CMakeFiles/doxymain.dir/build.make:175: generated_src/code.cpp] Segmentation fault (core dumped) make[2]: *** Deleting file 'generated_src/code.cpp' make[2]: Leaving directory '/tmp/doxygen_objdir' make[1]: *** [CMakeFiles/Makefile2:540: src/CMakeFiles/doxymain.dir/all] Error 2 make[1]: Leaving directory '/tmp/doxygen_objdir' make: *** [Makefile:182: all] Error 2 make: Leaving directory '/tmp/doxygen_objdir' ```

Again, if I switch to flex 2.6.3, it builds fine.

[albert-github]

1. Did you build flex 2.6.3 yourself ?
2. With which compiler did you create the 2.6.4 flex executable?

In your output I see:

• [ 23%] [FLEX][configimpl] Building scanner with flex 2.6.4 and later that the file configimpl.cpp.o is build, so in principle flex can work (same accounts for the earlier xml.l and mscgen_lexer.l
• the warnings about %name-prefix can be ignored.
• the code.l crashes flex, this is strange. I don't directly see the reason.
  • Might have to do with the installed system libraries and the needed libraries (hence my questions at the top). Maybe best to do ldd flex for the 2.6.3 and the 2.6.4 version
  • Run further with make -i so we can see what happens with the other*.l` files (maybe we see a pattern, though I'm not optimistic).
• You mention the --disable-shared is required due to some other issue in flex: what issue?

[lyu]

Yes, both flex 2.6.3 and 2.6.4 were built by me, with the identical configuration shown above.

Both flex, doxygen, and everything under $HOME/.local were built with GCC 10.2.0.

flex 2.6.3

[login1 ~]$ ldd .local/bin/flex
    linux-vdso.so.1 (0x0000ffffbe7b0000)
    libm.so.6 => /lib64/libm.so.6 (0x0000ffffbe6b0000)
    libc.so.6 => /lib64/libc.so.6 (0x0000ffffbe530000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffffbe7c0000)

flex 2.6.4

[login1 ~]$ ldd .local/bin/flex
    linux-vdso.so.1 (0x0000ffffbe7b0000)
    libm.so.6 => /lib64/libm.so.6 (0x0000ffffbe6b0000)
    libc.so.6 => /lib64/libc.so.6 (0x0000ffffbe530000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffffbe7c0000)

Build log with make -i: log.txt

The issue with --disable-shared is that the dynamic library of flex breaks configure for some software projects, so some distros explicitly disables the dynamic library of flex. For example, Fedora and CentOS does not install libfl.so when you install the default flex package, and Gentoo explicitly disables the shared library in their build scripts. If you are interested, here are the references: ref1, ref2, ref3, ref4.

[albert-github]

This remains very strange,. I now don't see the segmentation fault anymore (might have to do with not disabling the nls support).

• looks like you start each build from scratch, that is good for the tests.
• a strange thing is the /lustre/home/user/.local/bin/m4:stdin:679: ERROR: end of file in string though might give a clue as well (in my opinion a bit strange that m4 is called here )
• can you attach the generated code.cpp and scanner.cpp when it is generated with 2.6.3 and with 2.6.4 (also the log file from the 2.6.3 build would be nice).

[lyu]

Here you go: info.tar.gz

[albert-github]

Thanks for the information.

• Compared the the first run we don't see the segmentation fault anymore (might be due to the compilation without the --disable-nls
• the files code.cpp and scanner.cpp are truncated in the "middle" of the file. The truncation is not at a buffer boundary (1024 or 2048 or ... bytes) so looks like this is all that flex produced.
  • looks like this directly has to do with the messages:

    /lustre/home/user/.local/bin/m4:stdin:679: ERROR: end of file in string
    /lustre/home/user/.local/bin/m4:stdin:728: ERROR: end of file in string

    which are only in the log_264. But it is strange, it looks like that this is only shown with doctokenizer.l and scanner.l and doctokenizer.l looks like to compile and some other *.l don't compile. I don't know what is fed into m4 by flex an what cases this (haven't see this before).

  • do the generated scanners (e.g. xmlcode.cpp) that compile successfully end with something like:

    #if USE_STATE2STRING
    #include "xmlcode.l.h"
    #endif

    otherwise it might be sheer luck that they compiled.

• worrying for me is also the warning:

  dev/shm/doxygen_src/src/regex.cpp: In function 'bool reg::isalpha(char)':
  /dev/shm/doxygen_src/src/regex.cpp:40:11: warning: comparison is always false due to limited range of data type [-Wtype-limits]
   40 |   return c<0 || (c>='a' && c<='z') || (c>='A' && c<='Z');
      |          ~^~

  char is according to my knowledge signed and thus can be negative. Is there any setting on your system that causes this warning (e.g. setting of a compilation flag or setting of the locale)?

Did you have a look at the source repository of flex (https://github.com/westes/flex)? I see here (quickly) some issues that might be of interest:

• https://github.com/westes/flex/issues/339
  • Note with doxygen you can give make LEX_FLAGS=-d, this will probably also work with make LEX_FLAGS=--noline
• https://github.com/westes/flex/issues/453
• https://github.com/westes/flex/issues/469

All in all a very strange problem. I think there might be other interesting issues as well, see e.g. https://github.com/westes/flex/issues?q=is%3Aissue+is%3Aopen+m4.

[doxygen]

* worrying for me is also the warning:
  ```
  dev/shm/doxygen_src/src/regex.cpp: In function 'bool reg::isalpha(char)':
  /dev/shm/doxygen_src/src/regex.cpp:40:11: warning: comparison is always false due to limited range of data type [-Wtype-limits]
     40 |   return c<0 || (c>='a' && c<='z') || (c>='A' && c<='Z');
        |          ~^~
  ```

  `char` is according to my knowledge `signed`  and thus can be negative.

Turns out the signed-ness of char is actually implementation defined (so the above code is wrong on some platforms ☹️ ), see also https://www.linuxtopia.org/online_books/an_introduction_to_gcc/gccintro_71.html. So it may be either signed or unsigned. For x86/gcc is it is signed unless -funsigned-char is used. So it is maybe interesting/relevant to know the platform that is being used (e.g. the output of uname -a).

[lyu]

@doxygen It's an AArch64 machine, Marvell ThunderX2.

[albert-github]

That is horrible (and that is mildly put), this might be one of the less known pitfalls.

I do see in the C++ standard indeed:

Type char is a distinct type that has an implementation-defined choice of “signed char” or “unsigned char” as its underlying type.

In the original K&R I don't see an explicit restriction, but in the 2nd edition (page 36) I see:

Whether plain chars are signed or unsigned is machine-dependent, but printable characters are always positive.

I also see in the C standard (and in the C89 there were similar wordings):

The three types char, signed char, and unsigned char are collectively called the character types. The implementation shall define char to have the same range, representation, and behavior as either signed char or unsigned char.

I see that gcc has:

  -fsigned-char

x Let the type "char" be signed, like "signed char".

      Note that this is equivalent to -fno-unsigned-char, which is the negative form of -funsigned-char.  Likewise, the option -fno-signed-char is equivalent to -funsigned-char.

• @doxygen should we add this to the compile flags for gcc?
• @lyu maybe you could try to compile flex and doxygen with this option.

[lyu]

I checked some other generated cpp files and they do end with that #if USE_STATE2STRING part.

Compiling both flex and doxygen with -fsigned-char in CFLAGS did get rid of the warning, but that's about it, doxygen still fails with the same error.

Compiling with LEX_FLAGS="-d --noline" however, leads to more m4 errors and fewer compilation errors.

Here is the log: noline.txt

[albert-github]

I'm not 100% sure whether the CFLAGS is the correct flag to be used or that one has to use CXXFLAGS or CPPFLAGS (or with an underscore). I think the first thing to concentrate on is to get flex produce the right code, i.e that all the files end with the #if USE_STATE2STRING part.

Note: In the noline.txt I don't see that code.cpp gives errors anymore.

[lyu]

I have added -fsigned-char to all the *FLAGS I can think of, no change.

If it is OK for you, I can try to create a QEMU image that reproduces the issues I am seeing right now so you can play with it yourself. I will probably need to wait till the weekend though. Sounds like a plan?

[albert-github]

To be honest I'm a bit out of ideas, so probably playing around will not help.

• did you post a question in the flex group / forum to see if they have some idea?
• @doxygen do you have more ideas?

[jannick0]

@lyu From far sight it appears that your local flex is broken and does not generate valid c-code for some files. So focus on flex rather.

Did you run make check for flex 2.6.4 to see if anything essential is missing and tests run through smoothly? You might want to share

• config.cache (created when -C added to the configure flags),
• config.log and
• the logs for make V=1 and make check V=1

[lyu]

@jannick0 I did run make check for flex and all 114 unit tests pass.

Here are the requested logs. flex_logs.tar.gz

[jannick0]

Thanks - this is not exactly what I expected, so flex seems to be OK. Next, let's run all .l files in the doxygen repo against your flex (bypassing cmake).

Could you please cd to the top repo dir doxygen and run from here the attached script check-flex.sh.txt (drop the extension .txt) which should create the archive flex-test.tar.gz. Then upload the archive whose size I hope will not be too big. Thanks.

[lyu]

@jannick0 I saw quite a few segfaults again while running your scripts. Here is the tarball :) flex-test.tar.gz

[jannick0]

@lyu Great - many thanks. It looks like the issue happens on a deterministic basis - for more than 10 .l files. It might well be that it depends on the size of the .l file such that some realloc causes the problem when flex increases the size of a string buffer.

I'll try to look into that tomorrow. Hoping I can chase things down on my system not knowing what is really happening on yours. :) Fingers crossed.

[jannick0]

OK - this took some work. Let's assume that code.l is a culprit file which does show the issue with your version flex-2.6.4 you used in our previous tests.

Please run the following steps, and stop if a step does not show the expected behaviour:

1. copy code.l from the doxygen repo to your flex-2.6.4 build tree
2. run flex-2.6.4 --preproc=0 -o code.tmp code.l. This should show a SEGFAULT as it did in our previous tests.
3. compile a special flex version with debug symbols by
   • adding ac_cv_func_reallocarray=no to your configure flags
   • building a flex debug version (I can do that by saying make V=1 CFLAGS=-gdwarf-2) and call the built executable, say, flex-2.6.4-dbg
   • please share config.cache, config.h, config.log, make-debug.log
4. run ./flex-2.6.4-dbg --preproc=0 -o code.tmp code.l (adjust path to code.l if needed - or simply copy the file). Does this still show the SEGFAULT? If so please continue with the next step, otherwise go to step 6.
5. create and share a backtrace log of the SEGFAULT of the previous step. With GDB I would do that as follows. (Not sure if GDB can handle longjmps the flex code uses and GDB can show us what happens here.)

   gdb --args ./flex-2.6.4-dbg --preproc=0 -o code.tmp code.l
   b exit
   run (... wait until breakpoint is hit)
   bt

6. run ./flex-2.6.4-dbg -o code.cpp code.l and check if code.cpp has valid c-code.

Let's see what the tests show. I hope that the step instructions are clear. Fingers crossed the result of the steps 4 and 6 are fine! ;)

[lyu]

Adding ac_cv_func_reallocarray=no did make the segfault disappear and the generated code.cpp looks flawless!

Here are the logs. logs.tar.gz

Edit: compiling doxygen with this 'special' flex also worked, hooray!

[jannick0]

Fantastic - all this is really the best outcome!!!

For the record here the conclusions:

• flex 2.6.4: the configure flag ac_cv_func_reallocarray=no makes flex call the ordinary function realloc instead of reallocarray if available on the system. Introduced by westes/flex@9c54eb6e30, first in flex 2.6.4.

• NOTE: The next flex release (>2.6.4) might prefer to use reallocarr if available on the system (over reallocarray over realloc) as introduced by westes/flex@2b290d8ebdfa. In case of a similar issue adding ac_cv_func_reallocarr=no might be a similar remedy.

@lyu It appears that reallocarray is buggy on your system. On my system (MSYS2) flex uses reallocarray which runs its testsuite without any error and generates valid cpp code for all .l files of the doxygen repo. You might want to further investigate and/or file a bug report with your system distro?

@doxygen It might be helpful to drop the release tarballs' dependency on flex and/or bison by shipping the generated c/cpp code files (instead of the .l/.y files) in the tarball. This would avoid any issues about flex and/or bison.

Thanks.

[albert-github]

@jannick0 Thanks for the thorough investigations.

@doxygen It might be helpful to drop the release tarballs' dependency on flex and/or bison by shipping the generated c/cpp code files (instead of the .l/.y files) in the tarball. This would avoid any issues about flex and/or bison.

My thoughts: Nice suggestion but I'm not sure whether this won't backfire in respect to development. The indicated problem so far only appeared once (as far as I know) most of the time the system / precompiled executables for flex / bison are used. The big question for me still is is this a configuration problem or an incorrect call to an allocation routine or a bug an the allocation routine. For some other files (in the vhdl directory using JavaCC) we do provide the original file and the generated files as by default not all systems have JavaCC (and java). With flex / bison I think the situation is a bit different, a lot of systems will have flex / bison installed as development tools so at that moment we would have again the same problem as in this issue.

[lyu]

The system I am having issues with (CentOS 8.1) has glibc 2.28, and the reallocarray function in stdlib.h is guarded by __USE_GNU. The manpage confirms this.

I wrote a very simple C program to test reallocarray, and noticed that if I don't add -D_GNU_SOURCE to the flags, gcc/clang only throws a warning about implicit declaration of reallocarray (the first search result of this warning on Google is literally this), and a warning about having to convert the int return type of reallocarray to void* (I have zero idea why). The same warnings are seen when I compile flex, perhaps it's just that the mysterious alternative reallocarray fooled the configure script, and the unit tests of flex didn't stress this function enough.

The resulting test program segfaults when I try to reallocarray 262144 bytes of memory, but not 131072 bytes (again, no idea why). Once I add -D_GNU_SOURCE no warnings are shown and the test program works fine.

I saw that this issue is patched in the development version of flex in the link above, so I will just wait for a new release of flex, and remember to add -D_GNU_SOURCE in the meantime.

@albert-github @jannick0 Thank you very much for your thorough investigation and timely replies! I've got my answer and please feel free to close this issue whenever you are done.

[albert-github]

So all in all my conclusion is hat this is not a doxygen issue, setting this issue to "invalid".

@lyu thanks for the last report. @jannick0 thanks for all the investigations

[jannick0]

@doxygen It might be helpful to drop the release tarballs' dependency on flex and/or bison by shipping the generated c/cpp code files (instead of the .l/.y files) in the tarball. This would avoid any issues about flex and/or bison.

My thoughts:

I am adding mine inline below for your consideration:

Nice suggestion but I'm not sure whether this won't backfire in respect to development.

Thorough development is mainly based on a git repository, but not a release tarball.

The indicated problem so far only appeared once (as far as I know) most of the time the system / precompiled executables for flex / bison are used.

flex <= 2.6.4 does not ship any test for the issue at hand, so it might be dormant in precompiled flex versions on systems with reallocarray.

The big question for me still is is this a configuration problem or an incorrect call to an allocation routine or a bug an the allocation routine.

Solved by using an appropriate cpp macro (like -D_GNU_SOURCE, see above). Thanks to @lyu .

For some other files (in the vhdl directory using JavaCC) we do provide the original file and the generated files as by default not all systems have JavaCC (and java). With flex / bison I think the situation is a bit different, a lot of systems will have flex / bison installed as development tools so at that moment we would have again the same problem as in this issue.

The bison team is going forward in developing new features and a new syntax, while as of now the prior syntax version is already tagged as deprecated and the bison user is encouraged to upgrade the .y file (sample in the doxygen repo below). The prior syntax will likely be removed in future bison releases and there might be a point in time that bison syntax is version dependent; this will - if it comes true - definitely be a nightmare, of course, when .y files with new bison syntax cannot be compiled on systems having lower bison versions. (Needless to say that I am not in favour of any non-downward-compatibility.) But this seems to be one point why GNU projects keep promoting the idea to keep release tarballs independent of development tools (autotools, m4, flex, bison) and ship code files generated by flex/bison. I believe this is a fair point.

$ bison --version
bison (GNU Bison) 3.7.6

$ cd /tmp/doxygen

$ bison -o constexp.c src/constexp.y
src/constexp.y:35.1-25: warning: deprecated directive: ‘%name-prefix "constexpYY"’, use ‘%define api.prefix {constexpYY}’ [-Wdeprecated]
   35 | %name-prefix "constexpYY"
      | ^~~~~~~~~~~~~~~~~~~~~~~~~
      | %define api.prefix {constexpYY}
src/constexp.y: warning: fix-its can be applied.  Rerun with option '--update'. [-Wother]

Happy if this issue will be closed.