search

zephyrproject-rtos / zephyr

Primary Git Repository for the Zephyr Project. Zephyr is a new generation, scalable, optimized, secure RTOS for multiple hardware architectures.
https://docs.zephyrproject.org
Apache License 2.0
10.49k stars 6.42k forks source link

correct libgcc not getting linked for CONFIG_FLOAT=y on ARM #2092

Closed zephyrbot closed 7 years ago

zephyrbot commented 8 years ago

Reported by Kumar Gala:

When enabling CONFIG_FLOAT=y we aren't linking in the correct libgcc.a on ARM architectures. There are a number of issues here:

  1. In the Zephyr toolchain the Makefile (scripts/Makefile.toolchain.zephyr) we need to set CROSS_COMPILE_arm_isa to 'fpu' if CONFIG_FLOAT is enabled.
  2. if we do #1, we run into a build/link issue:

/opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_powisf2.o) uses VFP register arguments, .tmp_zephyr.prebuilt does not /opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_powisf2.o) /opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_udivmoddi4.o) uses VFP register arguments, .tmp_zephyr.prebuilt does not /opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_udivmoddi4.o) collect2: error: ld returned 1 exit status

This is due to building the kernel with -mfloat-abi=softfp, while the Zephyr toolchain only supports -mfloat-abi=hard.

While we can quickly fixing this with some minor tweaks, are more proper fix would utilize -print-libgcc-file-name to get the libgcc path and we'd pickup:

https://gerrit.zephyrproject.org/r/#/c/2899/

To get multilib support for -mfloat-abi=softfp, and -mfloat-abi=hard

(Imported from Jira ZEP-555)

zephyrbot commented 8 years ago

by Juro Bystricky:

Yes, multilibs need some cleanup. For example, the following should be in Makefile.toolchain.zephyr:

{code:java} ifeq ($(ARCH),arm) CROSS_COMPILE_TARGET_arm = arm-poky-eabi CROSS_COMPILE_arm=$(TOOLCHAIN_HOME)/usr/bin/$(CROSS_COMPILE_TARGET_arm)/$(CROSS_COMPILE_TARGET_arm)- SYSROOT_arm := ${ZEPHYR_SDK_INSTALL_DIR}/sysroots/armv5-poky-eabi

libgcc

LIBGCC_DIR_arm = $(shell dirname $(CROSS_COMPILE_arm)gcc --sysroot=$(SYSROOT_arm) $(KBUILD_CFLAGS) -print-libgcc-file-name) LIB_INCLUDE_DIR_arm += -L ${LIBGCC_DIR_arm}

newlib lib/include

NEWLIB_DIR_arm = $(shell $(CROSS_COMPILE_arm)gcc --sysroot=$(SYSROOT_arm) $(KBUILD_CFLAGS) -print-multi-directory) LIB_INCLUDE_DIR_arm += -L $(SYSROOT_arm)/usr/lib/$(NEWLIB_DIR_arm)/ TOOLCHAIN_CFLAGS_arm = -I $(SYSROOT_arm)/usr/include endif

{code}

Similar setup for other multilib toolchains (ARC, Nios2, ...). Basically, we query the compiler itself which library should be used given compiler flags (options). One caveat: KBUILD_CFLAGS must be set, as it contains all GCC command line options. Currently Makefile.toolchain.zephyr is included before this is done, so some re-ordering is needed as well.

zephyrbot commented 8 years ago

by Kumar Gala:

Fixed by combination of SDK 0.8.2 (which supports multilibs for softfp and hard fp ABIs). Plus change to allow user to select which Floating Point ABI they want to build with.

zephyrbot commented 8 years ago

by Sharron LIU:

Please reporter verify and close this. Thanks.

zephyrbot commented 8 years ago

by Mark Linkmeyer:

Hi Kumar Gala , as the Reporter of this bug we need you to verify it's fixed since it's in the Resolved state. Please verify it ASAP so we know whether or not more work is required to fix it before the upcoming 1.5 release. If it's fixed, please move it to the Verified state. Thanks!

zephyrbot commented 8 years ago

by Marcus Shawcroft:

The underlying issue is still present with the SDK 0.8.2, but manifests when building with hard float disabled.

The multilib configuration for arm-poki-eabi-gcc is: .; thumb;@mthumb fpu;@mfloat-abi=hard thumb/thumb2;@mthumb@march=armv7

The KBUILD_CFLAGS options passed for an frdm_k64f build include: -mcpu=cortex-m4 -mthumb -march=armv7e-m

The architecture 'armv7e-m' is correct for a cortex-m4. This set of options will match the first multlib (ie thumb;@thumb). The libgcc.a file in that directory was built for armv4t (where thumb and arm state are legitimate). The armv7e-m architecture is thumb2 only, hence a binary linked against this first multilib is illegal.

For reference, there is comprehensive multilib support for the full range of ARM R and M profile architecture permutations provided in the the ARM/embedded-5 upstream GCC branch. See here for the first commit: https://gcc.gnu.org/viewcvs/gcc/branches/ARM/embedded-5-branch/gcc/config/arm/t-baremetal?revision=231750&view=markup

This patch provides support for the following multilibs: .; thumb;@mthumb fpu;@mfloat-abi=hard armv6-m;@mthumb@march=armv6s-m armv7-m;@mthumb@march=armv7-m armv7e-m;@mthumb@march=armv7e-m armv7-ar/thumb;@mthumb@march=armv7 armv8-m.base;@mthumb@march=armv8-m.base armv8-m.main;@mthumb@march=armv8-m.main armv7e-m/softfp/fpv5-sp-d16;@mthumb@march=armv7e-m@mfloat-abi=softfp@mfpu=fpv5-sp-d16 armv7e-m/softfp/fpv5-d16;@mthumb@march=armv7e-m@mfloat-abi=softfp@mfpu=fpv5-d16 armv7e-m/softfp;@mthumb@march=armv7e-m@mfloat-abi=softfp@mfpu=fpv4-sp-d16 armv7e-m/fpu/fpv5-sp-d16;@mthumb@march=armv7e-m@mfloat-abi=hard@mfpu=fpv5-sp-d16 armv7e-m/fpu/fpv5-d16;@mthumb@march=armv7e-m@mfloat-abi=hard@mfpu=fpv5-d16 armv7e-m/fpu;@mthumb@march=armv7e-m@mfloat-abi=hard@mfpu=fpv4-sp-d16 armv7-ar/thumb/softfp;@mthumb@march=armv7@mfloat-abi=softfp@mfpu=vfpv3-d16 armv7-ar/thumb/fpu;@mthumb@march=armv7@mfloat-abi=hard@mfpu=vfpv3-d16 armv8-m.main/softfp/fpv5-sp-d16;@mthumb@march=armv8-m.main@mfloat-abi=softfp@mfpu=fpv5-sp-d16 armv8-m.main/softfp/fpv5-d16;@mthumb@march=armv8-m.main@mfloat-abi=softfp@mfpu=fpv5-d16 armv8-m.main/fpu/fpv5-sp-d16;@mthumb@march=armv8-m.main@mfloat-abi=hard@mfpu=fpv5-sp-d16 armv8-m.main/fpu/fpv5-d16;@mthumb@march=armv8-m.main@mfloat-abi=hard@mfpu=fpv5-d16

While this is not complete, it does provide a viable lowest common denominator for all current arm aarch32 cortex-m/r architectures.

This patch/support has not yet been presented for inclusion in upstream GCC trunk, but my understanding is that this is likely in the near future.

w.r.t the immediate issue in zephyr, possible short term options include: -) Add multilibs to cover just the current issue of broken cortex-m4 / aarch32e-m -) Backport the relevant multilib patches from upstream gcc/embedded-5 to the zephyr sdk (these patches are ARM centric and hence are not likely to destabalize the compiler behaviour on the other architecture we care about in zephyr).

It's worth noting that folks currently working on cortex-m0 / m0+ support are going to hit the same issue (cortex-m0 == armv6-m), hence a minimum viable set of armv6-m multlibs will be required to enable their work.

zephyrbot commented 8 years ago

by Marcus Shawcroft:

This is still broken:

with 0.8.2. On frdm_k64f configured CONFIG_FP_SOFTABI an inappropriate libgcc.a is linked resulting in ARM state code being executed on arm7e-m

with CONFIG_FP_HARDABI we get a link error (one object uses VFP registers another does not).

See my previous comment above for more detail.

zephyrbot commented 8 years ago

by Kumar Gala:

What error are you seeing. How to reproduce?

zephyrbot commented 8 years ago

by Marcus Shawcroft:

Reproduce like this:

$ cat prj.conf CONFIG_STDOUT_CONSOLE=y CONFIG_FLOAT=y

$ cat src/main.c double f = 2.0;

int main () { printf ("hello\n"); f = f / 3; printf ("f=%f\n", f); printf ("world\n"); }

$ make BOARD=frdm_k64f

gives.... /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -T linker.cmd @zephyr.lnk -o .tmp_zephyr.prebuilt /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: .tmp_zephyr.prebuilt uses VFP register arguments, /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o) does not /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o)

Adding CONFIG_FP_SOFTABI:

Results in an apparently successful build. Howver the binary produced contains ARM state code. If you try to execute the resulting binary it will repeatedly fault.

For direct evidence of the issue: $ arm-none-eabi-objdump -d outdir/frdm_k64f/zephyr.elf | grep -A 10 aeabi_ddiv | head -n 10 00000958 <__aeabi_ddiv>: 958: e92d4070 push {r4, r5, r6, lr} 95c: e3a0c0ff mov ip, #255 ; 0xff 960: e38ccc07 orr ip, ip, #1792 ; 0x700 964: e01c4a21 ands r4, ip, r1, lsr #20 968: 101c5a23 andsne r5, ip, r3, lsr #20 96c: 1134000c teqne r4, ip 970: 1135000c teqne r5, ip 974: 0b00005e bleq af4 <__aeabi_ddiv+0x19c> 978: e0444005 sub r4, r4, r5

ie ARM state code.

zephyrbot commented 8 years ago

by Kumar Gala:

Not able to reproduce. I'm able to run with both CONFIG_FP_SOFTABI and CONFIG_FP_HARDABI without issues. I also don't see the compile issue with HARDABI. Can you build with V=1 and report the link lines you are seeing.

zephyrbot commented 8 years ago

by Marcus Shawcroft:

Its possible to demonstrate correct behaviour here using the pre-built toolchain from https://launchpad.net/gcc-arm-embedded

$ arm-none-eabi-gcc -v ... gcc version 5.4.1 20160609 (release) ...

$ make BOARD=frdm_k64f CROSS_COMPILE=arm-none-eabi-

Generates binaries that execute correctly in both of the examples I gave above.

zephyrbot commented 8 years ago

by Marcus Shawcroft:

Hi, Link lines as request:

  rm -f libzephyr.a; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-ar rcTD libzephyr.a  drivers/built-in.o    lib/built-in.o  kernel/built-in.o  misc/built-in.o  net/built-in.o  boards/built-in.o  ext/built-in.o  usb/built-in.o  fs/built-in.o  arch/built-in.o
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -x assembler-with-cpp -nostdinc -undef -E -P \
     -D__GCC_LINKER_CMD__ -I/home/marcus/work/zephyr/zephyr-project/include \
    -I./include/generated  /home/marcus/work/zephyr/zephyr-project/arch/arm/soc/nxp_kinetis/k6x/linker.ld -o linker.cmd
  ( echo -nostartfiles -nodefaultlibs -nostdlib -static -Wl,-X -Wl,-N -Wl,--gc-sections -Wl,--build-id=none; echo "-Wl,-Map=zephyr.map"; echo "-L ./include/generated"; echo "-u _OffsetAbsSyms -u _ConfigAbsSyms"; echo "-e __start"; echo "-Wl,--start-group"; echo "-Wl,--whole-archive"; echo ""; echo " src/built-in.o"; echo "libzephyr.a"; echo "-Wl,--no-whole-archive"; echo "./arch/arm/core/offsets/offsets.o"; echo "-Wl,--end-group"; echo "-L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/thumb -L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/thumb  -lgcc"; ) > zephyr.lnk
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -T linker.cmd @zephyr.lnk -o .tmp_zephyr.prebuilt
  ( /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-objdump -S zephyr.elf > zephyr.lst; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-objcopy -S -O binary -R .note -R .comment -R COMMON -R .eh_frame zephyr.elf zephyr.bin; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-strip -s -o zephyr.strip zephyr.elf; )
zephyrbot commented 8 years ago

by Marcus Shawcroft:

Kumar, when run, does the example above print 0.0000 or something resembling 0.6 ?

zephyrbot commented 8 years ago

by Kumar Gala:

I saw 0.0000 once, but than starting seeing 0.6.

Can you try removing '-mfpu=fpv4-sp-d16' from arch/arm/Makefile and see what you get. I'm noticing that -mfloat-abi=hard -mfpu=fpv4-sp-d16 isn't getting me the code I'd expect (with either the SDK toolchain or the ARM embedded toolchain). However if I drop the -mfpu option things seem more inline with what I'd expect.

zephyrbot commented 8 years ago

by Marcus Shawcroft:

Configured for CONFIG_FP_HARDABI with '-mfpu=fpv4-sp-d16' from arch/arm/Makefile the link fails with:

   /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-ld -nostartfiles -nodefaultlibs -nostdlib -static   -r -o src/built-in.o src/main.o 
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -x assembler-with-cpp -nostdinc -undef -E -P \
     -D__GCC_LINKER_CMD__ -I/home/marcus/work/zephyr/zephyr-project/include \
    -I./include/generated  /home/marcus/work/zephyr/zephyr-project/arch/arm/soc/nxp_kinetis/k6x/linker.ld -o linker.cmd
  rm -f libzephyr.a; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-ar rcTD libzephyr.a  drivers/built-in.o    lib/built-in.o  kernel/built-in.o  misc/built-in.o  net/built-in.o  boards/built-in.o  ext/built-in.o  usb/built-in.o  fs/built-in.o  arch/built-in.o
  ( echo -nostartfiles -nodefaultlibs -nostdlib -static -Wl,-X -Wl,-N -Wl,--gc-sections -Wl,--build-id=none; echo "-Wl,-Map=zephyr.map"; echo "-L ./include/generated"; echo "-u _OffsetAbsSyms -u _ConfigAbsSyms"; echo "-e __start"; echo "-Wl,--start-group"; echo "-Wl,--whole-archive"; echo ""; echo " src/built-in.o"; echo "libzephyr.a"; echo "-Wl,--no-whole-archive"; echo "./arch/arm/core/offsets/offsets.o"; echo "-Wl,--end-group"; echo "-L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0 -L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/.  -lgcc"; ) > zephyr.lnk
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -T linker.cmd @zephyr.lnk -o .tmp_zephyr.prebuilt
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: .tmp_zephyr.prebuilt uses VFP register arguments, /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o) does not
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o)
zephyrbot commented 7 years ago

by Juro Bystricky:

Well, I think the problem is there are missing ARM libgcc.a libraries for each library variant. I am looking into it.

zephyrbot commented 7 years ago

by Juro Bystricky:

Confirmed. The binary is linked with incorrect libgcc.a (as it is the only one available - default tune). We have all libc.a (newlib) variants available, but libgcc.a variants are missing. Will be fixed in the next SDK (0.8.3 or 0.9)

zephyrbot commented 7 years ago

by Kumar Gala:

Where are missing, I see:

./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/thumb/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv6-m/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7-m/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/softfp/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/softfp/fpv5-d16/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/softfp/fpv5-sp-d16/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/fpu/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/fpu/fpv5-d16/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/armv7e-m/fpu/fpv5-sp-d16/libgcc.a ./sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a

zephyrbot commented 7 years ago

by Juro Bystricky:

Correct. All of the above will be present in the next SDK release.

zephyrbot commented 7 years ago

by Kumar Gala:

Sorry, wasn't clear. I was saying that was my list of libgcc.a from the 0.8.2 sdk install, so not sure what we are missing.

zephyrbot commented 7 years ago

by Juro Bystricky:

Argh, was looking at the next SDK under development. Yes, the libraries are present in SDK 0.8.2.

I was not able to reproduce the problem, but I have seen the link error while running sanitycheck (test_fp_shared for ARM). For some reason I don't see it anymore. (...uses VFP register arguments,...).

Marcus, just being paranoid: did you run "make pristine" after changing prj.conf? Can you check the Zephyr distro commit? (I use commit 2aa754497dc9d09b95445719811939334f0cf66b)

zephyrbot commented 7 years ago

by Juro Bystricky:

Additionally, repeating the steps from above (disassembly), I see code that looks like thumb2 code: mix of 32 bit and 16 bit opcodes $OBJDUMP_ARM -d outdir/frdm_k64f/zephyr.elf | grep -A 10 aeabi_ddiv | head -n 10 > harddump.log

{code:java}

0000094c <__aeabi_ddiv>:
     94c:   b570        push    {r4, r5, r6, lr}
     94e:   f04f 0cff   mov.w   ip, <span>#</span>255   ; 0xff
     952:   f44c 6ce0   orr.w   ip, ip, <span>#</span>1792  ; 0x700
     956:   ea1c 5411   ands.w  r4, ip, r1, lsr <span>#</span>20
     95a:   bf1d        ittte   ne
     95c:   ea1c 5513   andsne.w    r5, ip, r3, lsr <span>#</span>20
     960:   ea94 0f0c   teqne   r4, ip
     964:   ea95 0f0c   teqne   r5, ip
     968:   f000 f8a7   bleq    aba <__aeabi_ddiv+0x16e>

{code}

zephyrbot commented 7 years ago

by Marcus Shawcroft:

Folks, I've looked at this again this morning with fresh eyes and concluded that I botched my original analysis. It looks like I installed 8.2 but continued to execute 8.1.

With 8.2, I do see sane behaviour, my test app runs as expected.

Juro, Kumar, sorry for the distraction.

zephyrbot commented 7 years ago

by Juro Bystricky:

All's well that ends well. A simple FP test like this should be in the test suite though (maybe it is already).

zephyrbot commented 7 years ago

by Mark Linkmeyer:

Correcting the priority field