technyon / Arduino-CMake-Toolchain

CMake toolchain for all Arduino compatible boards
MIT License
20 stars 7 forks source link

Linking problems with the generated libraries -- is it possible to just use object files? #7

Open myklemykle opened 4 months ago

myklemykle commented 4 months ago

I am hitting the weirdest problem trying to use this package with the GCC ARM tools & Philhower RP2040 core. I just want to ask: is it possible to configure Arduino-CMake-Toolchain to build the final .elf/.uf2 file from separate object files, like arduino-cli does, instead of bundling them into libraries first? Because as far as I can tell, I'm hitting some kind of compiler bug or other problem when I link from libraries via cmake, but when I link from objects with arduino-cli everything is kosher.

So that's my question/issue for this repo. Below is a long, detailed description of the problem that maybe doesn't belong in this repo. I dunno yet if it's GCC's problem or Adafruit's problem or Earl Philhower's problem, but I just want to document it somewhere for the moment. Maybe the solution is obvious to other people.


I'm using an M1 Mac to compile an Arduino sketch for the RP2040 MCU. I'm using these GCC-ARM versions, provided by the Homebrew project:

 » arm-none-eabi-gcc --version
arm-none-eabi-gcc (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 13.2.1 20231009
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 » arm-none-eabi-ld -v        
GNU ld (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 2.41.0.20231009

This is an example sketch that shows the problem. It's just the basic Blink example, plus opening a USB serial connection and writing a message. I'm using the Adafruit-TinyUSB library option of the core (as opposed to the picousb option) so USE_TINYUSB is defined.

#ifdef USE_TINYUSB
#include <Adafruit_TinyUSB.h>
#endif

// the setup function runs once when you press reset or power the board
void setup() {
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
  // initialize USB Serial
  Serial.begin(115200);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);  // turn the LED on (HIGH is the voltage level)
  delay(1000);                      // wait for a second
  digitalWrite(LED_BUILTIN, LOW);   // turn the LED off by making the voltage LOW
  delay(1000);                      // wait for a second
  Serial.println("boop");
}

The code compiles and runs correctly with arduino-cli. With the build-system generated by Arduino-CMake-Toolchain, the code compiles and runs, the blinking happens correctly, but the serial output does not.

Examining the build artifacts, it looks like the .elf output of the Arduino-CMake build is missing some symbols (TinyUSB_Device_Init for one) that initialize the Adafruit-TinyUSB library. Here's the arduino-cli compiled .elf:

» arm-none-eabi-readelf -s abuild/Blink.ino.elf | grep TinyUSB
    69: 00000000     0 FILE    LOCAL  DEFAULT  ABS Adafruit_TinyUSB[...]
   152: 00000000     0 FILE    LOCAL  DEFAULT  ABS Adafruit_TinyUSB[...]
  1495: 10003af1    20 FUNC    GLOBAL DEFAULT    4 _Z21TinyUSB_Port[...]
  1539: 10003b11    40 FUNC    WEAK   DEFAULT    4 TinyUSB_Device_Task
  1656: 10003215    20 FUNC    WEAK   DEFAULT    4 TinyUSB_Device_Init
  1670: 10003aad    68 FUNC    GLOBAL DEFAULT    4 _Z23TinyUSB_Port[...]
  1733: 10003229    32 FUNC    WEAK   DEFAULT    4 TinyUSB_Device_F[...]
  1802: 10003b05    10 FUNC    GLOBAL DEFAULT    4 _Z28TinyUSB_Port[...]
  1891: 200013f8   404 OBJECT  GLOBAL DEFAULT   11 TinyUSBDevice

And here's the cmake-compiled .elf:

» arm-none-eabi-readelf -s cbuild/Blink.elf | grep TinyUSB 
   233: 00000000     0 FILE    LOCAL  DEFAULT  ABS Adafruit_TinyUSB[...]
  1416: 10004111    20 FUNC    GLOBAL DEFAULT    4 _Z21TinyUSB_Port[...]
  1459: 10004131    40 FUNC    WEAK   DEFAULT    4 TinyUSB_Device_Task
  1708: 10004125    10 FUNC    GLOBAL DEFAULT    4 _Z28TinyUSB_Port[...]
  1795: 20001328   404 OBJECT  GLOBAL DEFAULT   11 TinyUSBDevice

... and in fact, in the cmake-compiled version of the core main() routine, ld has actually replaced an unresolved link to <TinyUSB_Device_Init> with a nop instead of reporting any kind of link failure. Should ld ever do such a thing?

TinyUSB_Device_Init is called from main.cpp in the Philhower RP2040 Arduino core. This is the relevant code section with some undefined #ifdefs removed:

#ifdef USE_TINYUSB
    TinyUSB_Device_Init(0);
#endif

Here's the relevant section from the (working) arduino-cli version, obtained with arm-none-eabi-objdump -D -S abuild/Blink.ino.elf > Blink.ino.asm:

#ifdef USE_TINYUSB
    TinyUSB_Device_Init(0);
10007bd8: 2000        movs  r0, #0
10007bda: f7fb fb1b   bl  10003214 <TinyUSB_Device_Init>
#endif

and the compiled code of <TinyUSB_Device_Init> is also present in that .elf file:

10003214 <TinyUSB_Device_Init>:

//--------------------------------------------------------------------+
// Device
//--------------------------------------------------------------------+
#if CFG_TUD_ENABLED
void TinyUSB_Device_Init(uint8_t rhport) {
10003214: b510        push  {r4, lr}
  // Init USB Device controller and stack
  TinyUSBDevice.begin(rhport);
10003216: 4b03        ldr r3, [pc, #12] @ (10003224 <TinyUSB_Device_Init+0x10>)
void TinyUSB_Device_Init(uint8_t rhport) {
10003218: 0001        movs  r1, r0
  TinyUSBDevice.begin(rhport);
1000321a: 0018        movs  r0, r3
1000321c: f000 fa6c   bl  100036f8 <_ZN20Adafruit_USBD_Device5beginEh>
}
10003220: bd10        pop {r4, pc}
10003222: 46c0        nop     @ (mov r8, r8)
10003224: 200013f8  strdcs  r1, [r0], -r8

10003228 <TinyUSB_Device_FlushCDC>:
  tud_task();
}
#endif

That's all as you'd expect. But in the CMake-compiled version, obtained with arm-none-eabi-objdump -D -S cbuild/Blink.elf > Blink.asm, the same section is compiled like this:

#ifdef USE_TINYUSB
    TinyUSB_Device_Init(0);
100034b0: 2000        movs  r0, #0
100034b2: e000        b.n 100034b6 <main+0x9a>
100034b4: bf00        nop
#endif

... which means the call is just silently skipped. And the compiled code of <TinyUSB_Device_Init> is not present at all.

In both builds USE_TINYUSB was defined for every compile step. (I tried removing the #ifdef clause to force inclusion, with the same result.)

In both builds, main.cpp compiles correctly but unlinked. arduino-cli leaves it in make.cpp.o, while the cmake build places it in the library lib_arduino_lib_core.a, but the compiled code is the same:

    TinyUSB_Device_Init(0);
  72: 2000        movs  r0, #0
  74: f7ff fffe   bl  0 <TinyUSB_Device_Init>

In both builds, TinyUSB_Device_Init() compiles correctly but unlinked. cmake puts in in a library, lib_arduino_lib_Adafruit_TinyUSB.a, arduino-cli has it in Adafruit_TinyUSB_API.cpp.o, but the code is identical:

Disassembly of section .text.TinyUSB_Device_Init:

00000000 <TinyUSB_Device_Init>:

//--------------------------------------------------------------------+
// Device
//--------------------------------------------------------------------+
#if CFG_TUD_ENABLED
void TinyUSB_Device_Init(uint8_t rhport) {
   0: b510        push  {r4, lr}
  // Init USB Device controller and stack
  TinyUSBDevice.begin(rhport);
   2: 4b03        ldr r3, [pc, #12] ; (10 <TinyUSB_Device_Init+0x10>)
void TinyUSB_Device_Init(uint8_t rhport) {
   4: 0001        movs  r1, r0
  TinyUSBDevice.begin(rhport); 
   6: 0018        movs  r0, r3
   8: f7ff fffe   bl  0 <_ZN20Adafruit_USBD_Device5beginEh>
}
   c: bd10        pop {r4, pc}
   e: 46c0        nop     ; (mov r8, r8) 
  10: 00000000  andeq r0, r0, r0 

... and the other symbols all match as well. The code is all there, it's compiled, it's just not linked correctly.

Obviously both toolchains are eventually just calling arm-none-eabi-g++ and the other ARM-gcc tools. They appear to be calling them with all the same flags. This is the final link command with arduino-cli (super-long, gack, sorry):

/Volumes/External/mykle/Library/Arduino15/packages/rp2040/tools/pqt-gcc/2.2.0-d04e724/bin/arm-none-eabi-g++ -L/Volumes/External/mykle/Documents/Arduino/Blink/abuild -Werror=return-type -Wno-psabi -DUSE_TINYUSB -I/Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/libraries/Adafruit_TinyUSB_Arduino/src/arduino -DCFG_TUSB_MCU=OPT_MCU_RP2040 -DUSBD_PID=0x812d -DUSBD_VID=0x239a -DUSBD_MAX_POWER_MA=250 "-DUSB_MANUFACTURER=\"Adafruit\"" "-DUSB_PRODUCT=\"Feather RP2040 RFM\"" -DPICO_CYW43_ARCH_THREADSAFE_BACKGROUND=1 -DCYW43_LWIP=1 -DLWIP_IPV6=0 -DLWIP_IPV4=1 -DLWIP_IGMP=1 -DLWIP_CHECKSUM_CTRL_PER_NETIF=1 "-DARDUINO_VARIANT=\"adafruit_feather_rfm\"" -DTARGET_RP2040 -DPICO_FLASH_SIZE_BYTES=8388608 -march=armv6-m -mcpu=cortex-m0plus -mthumb -ffunction-sections -fdata-sections -fno-exceptions -DARM_MATH_CM0_FAMILY -DARM_MATH_CM0_PLUS -Os -u _printf_float -u _scanf_float @/Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/platform_wrap.txt -Wl,--cref -Wl,--check-sections -Wl,--gc-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--script=/Volumes/External/mykle/Documents/Arduino/Blink/abuild/memmap_default.ld -Wl,-Map,/Volumes/External/mykle/Documents/Arduino/Blink/abuild/Blink.ino.map -o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/Blink.ino.elf -Wl,--no-warn-rwx-segments -Wl,--start-group /Volumes/External/mykle/Documents/Arduino/Blink/abuild/sketch/Blink.ino.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/Adafruit_TinyUSB_API.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/Adafruit_USBD_CDC.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/Adafruit_USBD_Device.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/Adafruit_USBD_Interface.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/Adafruit_USBH_Host.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/cdc/Adafruit_USBH_CDC.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/hid/Adafruit_USBD_HID.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/midi/Adafruit_USBD_MIDI.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/msc/Adafruit_USBD_MSC.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/msc/Adafruit_USBH_MSC.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/ports/esp32/Adafruit_TinyUSB_esp32.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/ports/nrf/Adafruit_TinyUSB_nrf.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/ports/rp2040/Adafruit_TinyUSB_rp2040.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/ports/samd/Adafruit_TinyUSB_samd.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/video/Adafruit_USBD_Video.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/arduino/webusb/Adafruit_USBD_WebUSB.cpp.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/audio/audio_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/bth/bth_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/cdc/cdc_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/cdc/cdc_host.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/dfu/dfu_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/dfu/dfu_rt_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/hid/hid_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/hid/hid_host.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/midi/midi_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/msc/msc_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/msc/msc_host.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/net/ecm_rndis_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/net/ncm_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/usbtmc/usbtmc_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/vendor/vendor_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/class/video/video_device.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/common/tusb_fifo.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/device/usbd.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/device/usbd_control.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/host/hub.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/host/usbh.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/analog/max3421/hcd_max3421.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/microchip/samd/dcd_samd.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/nordic/nrf5x/dcd_nrf5x.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/raspberrypi/pio_usb/dcd_pio_usb.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/raspberrypi/pio_usb/hcd_pio_usb.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/raspberrypi/rp2040/dcd_rp2040.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/raspberrypi/rp2040/hcd_rp2040.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/raspberrypi/rp2040/rp2040_usb.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/portable/synopsys/dwc2/dcd_dwc2.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/Adafruit_TinyUSB_Arduino/tusb.c.o /Volumes/External/mykle/Documents/Arduino/Blink/abuild/libraries/SPI/SPI.a /Volumes/External/mykle/Documents/Arduino/Blink/abuild/core/core.a /Volumes/External/mykle/Documents/Arduino/Blink/abuild/boot2.o /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/ota.o /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/libpico.a /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/libpicow-noipv6-nobtc-noble.a /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/libbearssl.a -lm -lc -lstdc++ -lc -Wl,--end-group

And here is the final link step with cmake (shorter but still horribly long):

/Volumes/External/mykle/Library/Arduino15/packages/rp2040/tools/pqt-gcc/2.2.0-d04e724/bin/arm-none-eabi-g++  -L/Volumes/External/mykle/Documents/Arduino/Blink/cbuild -Werror=return-type -Wno-psabi  -DUSE_TINYUSB -I/Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/libraries/Adafruit_TinyUSB_Arduino/src/arduino -DCFG_TUSB_MCU=OPT_MCU_RP2040 -DUSBD_PID=0x812d -DUSBD_VID=0x239a -DUSBD_MAX_POWER_MA=250 "-DUSB_MANUFACTURER=\"Adafruit\"" "-DUSB_PRODUCT=\"Feather RP2040 RFM\"" -DPICO_CYW43_ARCH_THREADSAFE_BACKGROUND=1 -DCYW43_LWIP=1 -DLWIP_IPV6=0 -DLWIP_IPV4=1 -DLWIP_IGMP=1 -DLWIP_CHECKSUM_CTRL_PER_NETIF=1 "-DARDUINO_VARIANT=\"adafruit_feather_rfm\"" -DTARGET_RP2040 -DPICO_FLASH_SIZE_BYTES=8388608 -march=armv6-m -mcpu=cortex-m0plus -mthumb -ffunction-sections -fdata-sections -fno-exceptions  -DARM_MATH_CM0_FAMILY -DARM_MATH_CM0_PLUS  -Os -u _printf_float -u _scanf_float  @/Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/platform_wrap.txt -Wl,--cref -Wl,--check-sections -Wl,--gc-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--script=/Volumes/External/mykle/Documents/Arduino/Blink/cbuild/memmap_default.ld -Wl,-Map,/Volumes/External/mykle/Documents/Arduino/Blink/cbuild/Blink.map -o Blink.elf -Wl,--no-warn-rwx-segments -Wl,--start-group CMakeFiles/Blink.dir/Blink.ino.cpp.o  lib_arduino_lib_core.a lib_arduino_lib_Adafruit_TinyUSB.a lib_arduino_lib_SPI.a lib_arduino_lib_SdFat.a lib_arduino_lib_Adafruit_TinyUSB.a lib_arduino_lib_SPI.a lib_arduino_lib_SdFat.a lib_arduino_lib_USB.a lib_arduino_lib_FatFSUSB.a lib_arduino_lib_FatFS.a lib_arduino_lib_FS.a lib_arduino_lib_FatFS.a lib_arduino_lib_FS.a lib_arduino_lib_ff.a lib_arduino_lib_AudioBufferManager.a lib_arduino_lib_FreeRTOS.a lib_arduino_lib_core.a   /Volumes/External/mykle/Documents/Arduino/Blink/cbuild/boot2.o /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/ota.o  /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/libpico.a /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/libpicow-noipv6-nobtc-noble.a  /Volumes/External/mykle/Library/Arduino15/packages/rp2040/hardware/rp2040/3.9.2/lib/libbearssl.a -lm -lc -lstdc++ -lc -Wl,--end-group

The only significant difference i can find in the build output between the two processes is that the cmake process first bundles the objects into libraries, and then does the final link against those libraries, while the arduino-cli process compiles a zillion separate object files and then does the final link against the entire list of them.

That shouldn't make a difference in the compiled code, right?