Internal compiler errors when enabling LTO

[DurandA]

I got the following error when compiling the LiteX bios when enabling LTO:

 CC       bios.elf
during RTL pass: expand
/home/duranda/devel/liteboot/venv/lib/python3.8/site-packages/litex/soc/software/libbase/libc.c: In function 'skip_atoi':
/home/duranda/devel/liteboot/venv/lib/python3.8/site-packages/litex/soc/software/libbase/libc.c:418:5: internal compiler error: in insert_value_copy_on_edge, at tree-outof-ssa.c:334
  418 | int skip_atoi(const char **s)
      |     ^
0x583a9f insert_value_copy_on_edge
    ../.././riscv-gcc/gcc/tree-outof-ssa.c:334
0x583a9f eliminate_phi
    ../.././riscv-gcc/gcc/tree-outof-ssa.c:785
0x583a9f expand_phi_nodes(ssaexpand*)
    ../.././riscv-gcc/gcc/tree-outof-ssa.c:1024
0x61a2ce execute
    ../.././riscv-gcc/gcc/cfgexpand.c:6520
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
lto-wrapper: fatal error: riscv64-unknown-elf-gcc returned 1 exit status
compilation terminated.
/home/duranda/riscv/lib/gcc/riscv64-unknown-elf/10.2.0/../../../../riscv64-unknown-elf/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

I am unfamiliar with GCC internals so please tell me if I need to submit a preprocessed source. I am using a freshly compiled GCC from master.

[jim-wilson]

If this is a link time LTO problem, then having preprocessed source for one file won't help. We would need all of the binaries being linked together. Ot try doing our own build from scratch. if this is a compile time LTO problem then preprocessed source would he helpful. The build log doesn't actually say what the compiler/linker command is, it is just an ambiguous "cc bios.org" that could be either a compile problem or a link problem, but I'm guessing a link problem.

If this is a link tiem LTO problem, it will probably be time consuming to investigate, and we may not have time to do that anytime soon.

[kito-cheng]

@DurandA could you give more information about how to reproduce this issue? maybe the steps to build the project?

[DurandA]

I upgraded the compiled code (LiteX bios) to the latest revision and it produced a slightly different error (still related to lto_wrapper):

riscv64-unknown-elf-gcc -std=gnu99 -nostdlib -nodefaultlibs -Os -L/home/ubuntu/build/sim/software/include -T /home/ubuntu/.local/lib/python3.8/site-packages/litex/soc/software/bios/linker.ld -N -o bios.elf \
    ../libbase/crt0.o \
    isr.o boot-helper.o boot.o helpers.o cmd_bios.o cmd_mem.o cmd_boot.o cmd_i2c.o cmd_spiflash.o cmd_litedram.o cmd_liteeth.o cmd_litesdcard.o main.o complete.o readline.o \
    -L../libcompiler_rt \
    -L../libbase \
    -L../liblitedram \
    -L../libliteeth \
    -L../liblitespi \
    -L../liblitesdcard \
    -llitedram -lliteeth -llitespi -llitesdcard -lbase-nofloat -lcompiler_rt
during GIMPLE pass: fre
/home/ubuntu/.local/lib/python3.8/site-packages/litex/soc/software/bios/boot.c: In function 'serialboot.part.0':
/home/ubuntu/.local/lib/python3.8/site-packages/litex/soc/software/bios/boot.c:148:5: internal compiler error: in eliminate_stmt, at tree-ssa-sccvn.c:5971
  148 | int serialboot(void)
      |     ^
0x587213 eliminate_dom_walker::eliminate_stmt(basic_block_def*, gimple_stmt_iterator*)
    ../.././riscv-gcc/gcc/tree-ssa-sccvn.c:5971
0xb83add eliminate_dom_walker::before_dom_children(basic_block_def*)
    ../.././riscv-gcc/gcc/tree-ssa-sccvn.c:6399
0xb83add eliminate_dom_walker::before_dom_children(basic_block_def*)
    ../.././riscv-gcc/gcc/tree-ssa-sccvn.c:6332
0xf4b155 dom_walker::walk(basic_block_def*)
    ../.././riscv-gcc/gcc/domwalk.c:309
0xb7af3f eliminate_with_rpo_vn(bitmap_head*)
    ../.././riscv-gcc/gcc/tree-ssa-sccvn.c:6577
0xb8a0e2 do_rpo_vn
    ../.././riscv-gcc/gcc/tree-ssa-sccvn.c:7722
0xb8adc4 execute
    ../.././riscv-gcc/gcc/tree-ssa-sccvn.c:7804
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
lto-wrapper: fatal error: riscv64-unknown-elf-gcc returned 1 exit status
compilation terminated.
/home/ubuntu/riscv/lib/gcc/riscv64-unknown-elf/10.2.0/../../../../riscv64-unknown-elf/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

I enabled verbose make output so we can see each command. The complete log is available here.

Here are the steps on a clean system with Python3/pip in order to get the compilation running:

pip install git+https://github.com/DurandA/litex.git@enable-lto
pip install git+https://github.com/enjoy-digital/litedram.git
pip install git+https://github.com/enjoy-digital/liteeth.git
pip install git+https://github.com/enjoy-digital/liteiclink.git
pip install git+https://github.com/enjoy-digital/litescope.git
pip install git+https://github.com/litex-hub/pythondata-cpu-vexriscv.git

The bios will then be automatically compiled when using the litex_sim command. There is no need to install verilator as the compilation is done before the simulation.

[DurandA]

I downgraded GCC to 9.2.0 and yet another error. Here is the trace:

riscv64-unknown-elf-gcc -std=gnu99 -nostdlib -nodefaultlibs -Os -L/home/ubuntu/build/sim/software/include -T /home/ubuntu/.local/lib/python3.8/site-packages/litex/soc/software/bios/linker.ld -N -o bios.elf \
    ../libbase/crt0.o \
    isr.o boot-helper.o boot.o helpers.o cmd_bios.o cmd_mem.o cmd_boot.o cmd_i2c.o cmd_spiflash.o cmd_litedram.o cmd_liteeth.o cmd_litesdcard.o main.o complete.o readline.o \
    -L../libcompiler_rt \
    -L../libbase \
    -L../liblitedram \
    -L../libliteeth \
    -L../liblitespi \
    -L../liblitesdcard \
    -llitedram -lliteeth -llitespi -llitesdcard -lbase-nofloat -lcompiler_rt
during RTL pass: expand
/home/ubuntu/.local/lib/python3.8/site-packages/litex/soc/software/bios/readline.c: In function 'cread_add_char':
/home/ubuntu/.local/lib/python3.8/site-packages/litex/soc/software/bios/readline.c:153:13: internal compiler error: in expand_debug_locations, at cfgexpand.c:5456
  153 | static void cread_add_char(char ichar, int insert, unsigned long *num,
      |             ^
0x538093 expand_debug_locations
    ../.././riscv-gcc/gcc/cfgexpand.c:5456
0x538093 execute
    ../.././riscv-gcc/gcc/cfgexpand.c:6516
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
lto-wrapper: fatal error: riscv64-unknown-elf-gcc returned 1 exit status
compilation terminated.
/home/ubuntu/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

[kito-cheng]

TL;DR: Add CFLAGS to LDFLAGS can resolve that issue.

The reason why you got lots different strange internal compiler error is because all object files are compiled with -march=rv32im -mabi=ilp32, but during the linker stage, you don't give -march or -mabi option to gcc, so gcc using the default arch / abi to compile that - maybe -march=rv64gc + -mabi=lp64d, anyway it might different than your CFLAGS used when compiling those objects.

[DurandA]

Thanks @kito-cheng that was indeed the issue.