search

ARMmbed / mbed-os

Arm Mbed OS is a platform operating system designed for the internet of things
https://mbed.com
Other
4.67k stars 2.98k forks source link

IAR build failure when building mbed-os and examples #4879

Closed 0xc0170 closed 6 years ago

0xc0170 commented 7 years ago

Description

Bug

This command fails to compile (with traceback) (here is the job (internal link) http://mbed-ci-master-2.austin.arm.com:8081/job/examples_build_matrix/783/target=NUCLEO_F767ZI,toolchain=IAR ):

06:22:52 [DEBUG] Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_5a547c6d82611f64c4e2715b8a0e2ca1.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S
06:22:52 [DEBUG] Return: 3
06:22:52 Traceback (most recent call last):
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\make.py", line 280, in <module>
06:22:52     toolchain))
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\build_api.py", line 538, in build_project
06:22:52     objects = toolchain.compile_sources(resources, resources.inc_dirs)
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 923, in compile_sources
06:22:52     return self.compile_queue(queue, objects)
06:22:52   File "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 986, in compile_queue
06:22:52     raise ToolException(err)
06:22:52 ToolException
06:22:52 [mbed] Working path "C:\mj\workspace\bm_wrap\1063\examples\mbed-os-example-mesh-minimal" (program)

If you compare this command to other examples, the same command, for comparison:

Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_0f02e5fe9f3d74c786cba96e00bda7d5.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S
06:23:32 [DEBUG] Return: 0

For nucleo f429zi, it is different assembly file (here is the job (internal link) http://mbed-ci-master-2.austin.arm.com:8081/job/pr_pipeline/886/)

17:10:17 Compile [  0.2%]: cmain.S
17:10:17 [DEBUG] Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M4F -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M4 -DUSB_STM_HAL -DARM_MATH_CM4 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F429ZI\IAR\.includes_c45dbc904e060289cb07543bf5b1e267.txt -o .\BUILD\NUCLEO_F429ZI\IAR\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.o .\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.S
17:10:17 [DEBUG] Return: 3
17:10:17 Traceback (most recent call last):
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\make.py", line 280, in <module>
17:10:17     toolchain))
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\build_api.py", line 536, in build_project
17:10:17     objects = toolchain.compile_sources(resources, resources.inc_dirs)
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 909, in compile_sources
17:10:17     return self.compile_queue(queue, objects)
17:10:17   File "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal\mbed-os\tools\toolchains\__init__.py", line 972, in compile_queue
17:10:17     raise ToolException(err)
17:10:17 ToolException
17:10:17 [mbed] Working path "C:\mj\workspace\bm_wrap\953\examples\mbed-os-example-mesh-minimal" (program)

The same command (for sockets example):

17:10:20 Compile [  0.3%]: cmain.S
17:10:20 [DEBUG] Compile: C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm --cpu Cortex-M4F -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M4 -DUSB_STM_HAL -DARM_MATH_CM4 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F429ZI\IAR\.includes_7085507b6b69db011e3a2f64169c8a9d.txt -o .\BUILD\NUCLEO_F429ZI\IAR\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.o .\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.S
17:10:20 [DEBUG] Return: 0

I tried to reproduce locally, cant, was able to build 20x at least. The same IAR (Compared the version numbers for linker, assembler via these logs). Any suggestions how to reproduce this one?

Note, it happens randomly, not always in the CI.

Mine setup (IAR versions):

    IAR Assembler V7.80.4.12462/W32 for ARM
    Copyright 1999-2017 IAR Systems AB.

cc @theotherjimmy @studavekar

0xc0170 commented 7 years ago

cc @bcostm @adustm @LMESTM @jeromecoutant - you noticed these failures, but were not able to reproduce neither?

jeromecoutant commented 7 years ago

Hi Martin

I couldn't reproduce with these commands:

mbed import mbed-os-example-mesh-minimal cd mbed-os-example-mesh-minimal/mbed-os mbed compile -m NUCLEO_F429ZI -t IAR -v

Same IAR version: [DEBUG] Output: IAR ELF Linker V7.80.4.12462/W32 for ARM

Compile [ 0.2%]: cmain.S [DEBUG] Compile: C:/Program Files (x86)/IAR Systems/Embedded Workbench 7.5/arm\bin\iasmarm --cpu Cortex-M4F -DTRANSACTION_QUEUE_SIZE_SPI=2 -DCORTEX_M4 -DUSB_STM_HAL -DARM_MATH_CM4 -DFPU_PRESENT=1 -DUSBHOST_OTHER -DMBED_CMSIS_RTOS_CM -DCMSIS_RTOS -f .\BUILD\NUCLEO_F429ZI\IAR.includes_305dc107280526cdc975f1b97d3fea97.txt -o .\BUILD\NUCLEO_F429ZI\IAR\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.o .\mbed-os\cmsis\TOOLCHAIN_IAR\cmain.S [DEBUG] Return: 0

adustm commented 7 years ago

@jeromecoutant It looks like it is not easily reproducable. Did you notice such failures in our non regression test bench ? I can't remember of such an issue.

LMESTM commented 7 years ago

@jeromecoutant @adustm @0xc0170 ... I have never encountered this issue.

adbridge commented 7 years ago

@jeromecoutant @adustm @LMESTM do you guys have any way of putting those devices into a stress test that continually repeats the above steps 100s of times in a row to see if you hit the issue ?

LMESTM commented 7 years ago

@adbridge not sure to understand which devices you're referring to. It's a compilation issue that has only been seen so far on the mbnd ci test setup ... Would make more sense to investigate on this setup don't you think ?

adbridge commented 7 years ago

@LMESTM I meant running 100s of consecutive IAR compilations for the 2 devices mentioned and seeing if you can reproduce yourselves. We are investigating further in our CI as well.

LMESTM commented 7 years ago

@adbridge I couldn't find the "return 3" error code definition in IAR documentation, but I suspect a problem of access to the file rather than a problem of compilation itself (there is no compilation error here ...). Not clear to me how doing 100s of compilation here would help, @0xc0170 already did 20 without reproducing. Is it possible to restart the ci job and check if the problem happens again ?

studavekar commented 7 years ago

iar_crash

We are able to reproduce this failure now in CI and also locally however the procedure is bit cumbersome, We will try to have an script to reproduce the failure

Windows :

Failing case:

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

Passing case :

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

We have additionally did experiments which points it may be bug in IAR itself on handling file system paths.

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

Could not open file: 'C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN\startup_stm32f767xx.o'

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

    IAR Assembler V7.80.4.12462/W32 for ARM
    Copyright 1999-2017 IAR Systems AB.

Errors:   0
Warnings: 0
Bytes: 988

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.5\arm\bin\iasmarm" --cpu Cortex-M7 --fpu VFPv5 -DTRANSACTION_QUEUE_SIZE_SPI=2 -D__CORTEX_M7 -DARM_MATH_CM7 -D__FPU_PRESENT=1 -DUSBHOST_OTHER -D__MBED_CMSIS_RTOS_CM -D__CMSIS_RTOS -f .\BUILD\NUCLEO_F767ZI\IAR\.includes_db60bf2c33b9563ba7951b63c163c4e6.txt -o .\BUILD\NUCLEO_F767ZI\IAR\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.o .\mbed-os\targets\TARGET_STM\TARGET_STM32F7\TARGET_STM32F767xI\device\TOOLCHAIN_IAR\startup_stm32f767xx.S

    IAR Assembler V7.80.4.12462/W32 for ARM
    Copyright 1999-2017 IAR Systems AB.

Errors:   0
Warnings: 0
Bytes: 988

               * * *   I N T E R N A L   E R R O R   * * *

In function: Assembler
Diagnostic: non-initialized parameter
Line: 798   P0: 0    P1: 0

Dear User,
please send us a copy of the source file(s) that were assembled when
the internal error occurred.  Even better is if you could try to
pinpoint the problem with a small sample program.
In addition to the source file(s) we would like to receive information on:
  1. Error message (as above)
  2. Software version (see sign-on message)
  3. Command-line including any .xcl file used
  4. Hardware (memory etc.) and operating system version
  5. Where and when you bought the software
Please report this error to your distributor or the nearest IAR
Systems office. For further information please see the Software
Performance Report that can be found in your manual.

                           T H A N K   Y O U !

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>echo %errorlevel%
3

(.env) C:\mj\workspace\bm_wrap\1192\examples\mbed-os-example-mesh-minimal>
ciarmcom commented 6 years ago

ARM Internal Ref: MBOTRIAGE-196

0xc0170 commented 6 years ago

Fixed via #4977