mbed-os-example-client does not work on STM Nucleo F401RE

[markus-becker-tridonic-com]

I am trying to get the mbed-os-example-client to run on an STM Nucleo F401RE with the 6LoWPAN shield and the configuration.

In order to do so, I modified

diff --git a/mbed_app.json b/mbed_app.json
index 48fbd8d..f4f737b 100644
--- a/mbed_app.json
+++ b/mbed_app.json
@@ -2,7 +2,7 @@
     "config": {
         "network-interface":{
             "help": "options are ETHERNET,WIFI,MESH_LOWPAN_ND,MESH_THREAD",
-            "value": "ETHERNET"
+            "value": "MESH_LOWPAN_ND"
         },
         "wifi-ssid": {
             "help": "WiFi SSID",
@@ -15,9 +15,10 @@
     },
     "target_overrides": {
         "*": {
-            "target.features_add": ["IPV6", "CLIENT", "IPV4"],
+            "target.features_add": ["IPV6", "CLIENT"],
             "mbed-client.reconnection-count": 3,
-            "mbed-client.reconnection-interval": 5
+            "mbed-client.reconnection-interval": 5,
+            "nanostack.configuration": "lowpan_router"
         }
     }
-}
+}

and

diff --git a/tools/toolchains/gcc.py b/tools/toolchains/gcc.py
index 027a956..27dc11f 100644
--- a/tools/toolchains/gcc.py
+++ b/tools/toolchains/gcc.py
@@ -86,7 +86,7 @@ class GCC(mbedToolchain):
             self.flags["common"].append("-g")
             self.flags["common"].append("-O0")
         else:
-            self.flags["common"].append("-O2")
+            self.flags["common"].append("-Os")

         main_cc = join(tool_path, "arm-none-eabi-gcc")
         main_cppc = join(tool_path, "arm-none-eabi-g++")

so that it actually fits into ROM.

On the K64: mbed compile -c -t GCC_ARM -m K64F -j 4

Starting mbed Client example...
Using Mesh
Connecting to Mesh..
Connected to Network successfully
IP address ....
SOCKET_MODE : UDP
Connecting to coap://[...
Registered object successfully!

On the F401RE: mbed compile -c -t GCC_ARM -m NUCLEO_F401RE

Starting mbed Client example...
Using Mesh
Connecting to Mesh..
Connection to Network Failed -3009! Exiting application....

[markus-becker-tridonic-com]

@mlnx @bcostm

[yogpan01]

@SeppoTakalo @kjbracey-arm Can you please check this ?

[kjbracey]

Can we enable some trace?

[MarceloSalazar]

It seems tracing is not working as expected (ticket here https://github.com/ARMmbed/mbed-os/issues/449) Could we please help to investigate the issue?

[markus-becker-tridonic-com]

For which module should I enable tracing?

[kjbracey]

Simplest thing to do is add mbed-trace.enable to your mbed_app.json, in this section:

"target_overrides": {
    "*": {
        "mbed-trace.enable": 1
    }
}

[markus-becker-tridonic-com]

Debug output on F401RE:

<\n>
<\r>Connection to Network Failed -3009! Exiting application....<\r><\n>
<\0>Starting mbed Client example...<\r><\n>
Using Mesh<\r><\n>
<\n>
<\r>Connecting to Mesh..<\r><\n>
<\r><27>[2K<27>[90m[DBG ][core]: Allocate Root Tasklet<27>[0m<\r><\n>
<\r><27>[2K<27>[90m[DBG ][6lo ]: P.Init<\n>
<27>[0m<\r><\n>
<\r><27>[2K<27>[90m[DBG ][m6La]: init()<27>[0m<\r><\n>
<\r><27>[2K<27>[90m[DBG ][m6La]: connect()<27>[0m<\r><\n>
<\n>
<\r>Connection to Network Failed -3009! Exiting application....<\r><\n>

[kjbracey]

Did you clean after changing the config? (There's no autodependency on it). I'm expecting trace from the network stack.

[markus-becker-tridonic-com]

I assume so. My commandline was mbed compile -c -t GCC_ARM -m NUCLEO_F401RE -j 4.

[markus-becker-tridonic-com]

Somehow I suspect that the nd_tasklet_main does not start.

[kjbracey]

Seems likely, as you cleaned. It didn't manage to start the stack at all.

[markus-becker-tridonic-com]

Any hint on what could be going wrong?

[kjbracey]

Got to go now, but perhaps lack of memory for the Nanostack (nsdynmemlib) pool? Insufficent threads configured in RTX (I don't think error returns from thread creation are checked)?

[markus-becker-tridonic-com]

Any hints on what one can tweak and where?

[SeppoTakalo]

Way back, I recall seeing timing problems with atmel-rf-driver. I don't think we have tested with ST board for a while.

In order to verify actually what happened, we need to check with RF sniffer if the device is able to send anything at all.

There is also assumption on the driver that 'CS' release required specific timeouts, not sure if these are true on every platform. Please see: https://github.com/ARMmbed/atmel-rf-driver/blob/master/source/driverAtmelRFInterface.cpp#L175

[markus-becker-tridonic-com]

@SeppoTakalo Previously with mbed3 this was working on an ST board. Are there changes from mbed3 to mbed5 that could break it?

@kjbracey-arm Where could I start looking for nsdynmemlib and Insufficent threads? Highly likely that this is the source since the ST board has less RAM.

[markus-becker-tridonic-com]

Seems that the F401RE has "default_build": "small" in targets.json. This sets MBED_RTOS_SINGLE_THREAD which does not allow to start any threads. Shouldn't the inability to spawn new threads be caught? Does default_build small has other side effects?

When removing it, the example does not fit anymore into ROM. What has increased the ROM so much since mbed3?

[ciarmcom]

ARM Internal Ref: IOTCLT-879

[MarceloSalazar]

problem also reported during OOB RC#4

[bogdanm]

Should be fixed by https://github.com/ARMmbed/mbed-os-example-client/pull/58, needs verify.

[sg-]

What has increased the ROM so much since mbed3?

@markus-becker-tridonic-com using newlib which is thread safe versus newlib-nano which isn't. Its about a 20k ROM penalty with GCC

[markus-becker-tridonic-com]

Investigated more. mbedtls is a lot smaller now. Thanks.

The ETHERNET build is size-wise fine: 272724 bytes for K64F/GCC_ARM.

The MESH_LOWPAN_ND build is still big: 484478 bytes for NUCLEO_F401RE/GCC_ARM 484478 bytes for K64F/GCC_ARM

Now with the preparations for the public release, it seems you have put in the binary nanostack, with every option enabled (Thread + 6LoWPAN + BorderRouter?).

Is there a possibility to have a Thread Router, 6LoWPAN Router or Border Router nanostack binary and being able to select with e.g. NANOSTACK_BINARY_USER_CONFIG? If not, what is the supported way of putting in the nanostack source with the correct config selectable from the config NANOSTACK_USER_CONFIG_FILE like mbedtls does now with https://github.com/ARMmbed/mbed-os-example-client/pull/58?

[sg-]

@markus-becker-tridonic-com have a look at this. https://github.com/ARMmbed/mbed-os/blob/master/features/net/FEATURE_IPV6/CONTRIBUTING.md

[javier-moreno-tridonic-com]

@sg Thanks for the pointer!

Cloning sal-stack-nanostack-private to replace sal-stack-nanostack doesn't work without any tweaking (compilation gives a bunch of errors about type redefinitions, for instance). I'm trying to sort it out but I was wondering: Is there any specific branch/tag that we should use with the current mbed-os release? (so far I've tried master and the v5.0.5 tag)

[markus-becker-tridonic-com]

@MarceloSalazar would you have a look at @javier-moreno-tridonic-com's comment?

Do we need to clone all of

• mbed-mesh-api
• mbed-trace
• nanostack-hal-mbed-cmsis-rtos
• nanostack-libservice
• sal-stack-nanostack-eventloop
• sal-stack-nanostack-private

Or is just sal-stack-nanostack-private enough? Which tags/versions should work?

[hasnainvirk]

@markus-becker-tridonic-com @javier-moreno-tridonic-com

I just tried 5.0.5 and everything built smoothly. The version of Nanostack included in mbed-os is 5.0.5. It was checked out from branch 5.0-maint. Can you please have a Skype session session with me ? You don't need to clone anyting else. Just clone nanostack and checkout either v5.0.5 or 5.0-maint.
I tried with lowpan_router config:

+-----------------------------+--------+-------+-------+ | Module | .text | .data | .bss | +-----------------------------+--------+-------+-------+ | Fill | 692 | 26 | 83 | | Misc | 84121 | 2377 | 2936 | | features/FEATURE_CLIENT | 66627 | 11 | 53 | | features/FEATURE_COMMON_PAL | 20538 | 93 | 8404 | | features/frameworks | 3815 | 92 | 848 | | features/mbedtls | 75930 | 51 | 8755 | | features/net | 142494 | 302 | 34053 | | hal/common | 5657 | 20 | 365 | | hal/targets | 10739 | 4 | 1472 | | rtos/rtos | 451 | 4 | 0 | | rtos/rtx | 7037 | 20 | 2683 | | Subtotals | 418101 | 3000 | 59652 | +-----------------------------+--------+-------+-------+ Allocated Heap: unknown Allocated Stack: unknown Total Static RAM memory (data + bss): 62652 bytes Total RAM memory (data + bss + heap + stack): 62652 bytes Total Flash memory (text + data + misc): 421101 bytes

[hasnainvirk]

Had a Skype session with @javier-moreno-tridonic-com . The problem was the same file names on the include path. Its a known-issue on mbed-os5. When you have nanostack binaries you will not encounter this issue. However when you use sources, there are files like socket.h and thread.h whereas mbed-OS will bring in Socket.h and Thread.h. On unix based syatems you would still get away as they are case sensitive. Thread.h (mbed-os) and thread.h(nanostack) will be treated differently on unix. However, windows is case insensitive. Anyone building on windows with nanostack sources will encounter these issues. Conversely speaking, anyone adding a custom library with colliding names with any other module in the mbed-OS will suffer the saimilar fate. cc @markus-becker-tridonic-com

[javier-moreno-tridonic-com]

I opened a new issue for better tracking.

For completion, the compilation errors can be fixed in windows by replacing the following lines:

---

mbed-os

File mbed-os\features\FEATURE_CLIENT\mbed-client-classic\mbed-client-classic\m2mconnectionhandlerpimpl.h

Replace

#include "Socket.h"

With

#include "network-socket/Socket.h"

---

atmel-rf-driver

File source\driverAtmelRFInterface.cpp

Replace

#include "Mutex.h"
#include "Thread.h"
using namespace rtos;

With

#include "rtos.h"

[javier-moreno-tridonic-com]

New addition to this target issue. Using GCC is running great for us now. Today I tried to use the same code but compiled with armcc.

The example crashes with a HardFault in osEventObs->pre_start();

mbed-os/rtos/rtx/TARGET_CORTEX_M/rt_CMSIS.c

osStatus osKernelStart (void) {
  uint32_t stack[8];

  if (__get_IPSR() != 0U) {
    return osErrorISR;                          // Not allowed in ISR
  }

  /* Call the pre-start event (from unprivileged mode) if the handler exists
   * and the kernel is not running. */
  /* FIXME osEventObs needs to be readable but not writable from unprivileged
   * code. */
  if (!osKernelRunning() && osEventObs && osEventObs->pre_start) {
    osEventObs->pre_start(); //LINE CAUSING THE CRASH
  }

  switch (__get_CONTROL() & 0x03U) {
    case 0x00U:                                 // Privileged Thread mode & MSP
      __set_PSP((uint32_t)(stack + 8));         // Initial PSP
      if (os_flags & 1U) {                       
        __set_CONTROL(0x02U);                   // Set Privileged Thread mode & PSP
      } else {
        __set_CONTROL(0x03U);                   // Set Unprivileged Thread mode & PSP
      }
      __DSB();
      __ISB();
      break;
    case 0x01U:                                 // Unprivileged Thread mode & MSP
      return osErrorOS;
    case 0x02U:                                 // Privileged Thread mode & PSP
      if ((os_flags & 1U) == 0U) {              // Unprivileged Thread mode requested
        __set_CONTROL(0x03U);                   // Set Unprivileged Thread mode & PSP
        __DSB();
        __ISB();
      }
      break;
    case 0x03U:                                 // Unprivileged Thread mode & PSP
      if  (os_flags & 1U) { return osErrorOS; } // Privileged Thread mode requested
      break;
  }
  return __svcKernelStart();
}

Stack trace

mbed-os-example-client.elf

Thread #1 (Suspended : Signal : SIGINT:Interrupt)

HardFault_Handler() at startup_stm32f401xe.S:186 0x800051e  
<signal handler called>() at 0xfffffff9 
0x51520 
osKernelStart() at rt_CMSIS.c:572 0x8040a58 
__rt_entry() at RTX_CM_lib.h:689 0x8000338  

[javier-moreno-tridonic-com]

Any updates on this?

[yogpan01]

@SeppoTakalo @hasnainvirk Any progress on this issue ?

[hasnainvirk]

@javier-moreno-tridonic-com which version of armcc are you using ?

[javier-moreno-tridonic-com]

@hasnainvirk

This one

Product: MDK-ARM Standard Cortex-M only 5.20 (Flex) Component: ARM Compiler 5.06 update 2 (build 183) Tool: armcc [4d35cd]

[hasnainvirk]

@yogpan01 , @SeppoTakalo

Seemingly 96K RAM is not enough for mbed-os-example-client (with 6LoWPAN-ND) to work well.

i) With GCC, RAM gets consumed while mbedTLS starts writing ecp encryption tables in RAM. This is the error code MBEDTLS_ERR_SSL_ALLOC_FAILED -32512 (-0x7F00). But the system does not crash because with GCC there is a sane mechanism of Heap and stack overlap control. Anyway, app does not connect to mDS successfully because right after handshake, while writing some ecp stuff , memory gets exhausted.

ii) With armcc, RAM gets consumed just like th eprevious case. But there is no sane mechanism to keep check on heap and stack overlap. And hence you get the Hard fault.

An issue for sane handling of heap and stack collision is made on mbed OS https://github.com/ARMmbed/mbed-os/issues/2618.

However, we should now look how to reduce RAM consumption ? ROM Tables @SeppoTakalo @kjbracey-arm . Moreover, it seems we are low on ROM too as this platform have 512K of Flash.

[hasnainvirk]

@javier-moreno-tridonic-com Our Client Lead suggested a quick fix, https://github.com/ARMmbed/mbed-os-example-client/blob/master/mbedtls_mbed_client_config.h#L108 Change the undef to define , it will save 8K RAM for you at the expense of Flash. I tested it with ARMCC and GCC.

[javier-moreno-tridonic-com]

Good news, I'll try that ASAP.

On the meantime I made a test to check the RAM usage. To do so I tried modifying the startup_stm32f401xe.S to fill the RAM with a pattern.

; Reset handler
Reset_Handler    PROC
                 EXPORT  Reset_Handler             [WEAK]
        IMPORT  SystemInit
        IMPORT  __main

                 LDR     R0, =0x20000000  // lowest RAM position
                 LDR     R1, =0x20018000  // highest RAM position F401
                 SUBS    R1, R0           // RAM size
                 LDR     R2, =0xEFBEADDE  // fill pattern (DEADBEEF) in the dump file

FillRam          STR     R2, [R0]
                 ADDS    R0, R0, #4
                 SUBS    R1, #4
                 BNE     FillRam

                 LDR     R0, =SystemInit
                 BLX     R0
                 LDR     R0, =__main
                 BX      R0
                 ENDP

I dumped RAM contents on crash and used the following script to check how many 4 Bytes are different from this "DEADBEEF" pattern:

import argparse
import re

def main():
    parser = argparse.ArgumentParser(description='Counts used RAM from a RAM dump as plain text')
    parser.add_argument('-f', '--dumpfile', type=str, help='The dump file')
    args = parser.parse_args()

    dumpfile = open(args.dumpfile, 'rb')
    print dumpfile

    tokens = re.split(' |\n', dumpfile.read())
    usedram = 0

    print('Procesing ' + str(len(tokens)) + ' tokens...')
    for token in tokens:
        if token != 'DEADBEEF' and token != '':
            usedram += 4

    print('Used RAM: ' + str(usedram) + ' Bytes')

if __name__ == "__main__":
    main()

This was the result:

<open file 'F4RAMarmcc.dump', mode 'rb' at 0x028215A0>
Procesing 24576 tokens...
Used RAM: 61172 Bytes

I looks like on crash time there should be ~30KB free, assuming the startup file from armcc does the same as the one for gcc and fills with zeros the statically allocated variables. If that is not the case and the memory is not zeroed, it might be that the heap for the nanostack is taking those 30KB.

Please keep in mind that is the first time that I fiddle with this kind of assembler so there could be errors, I'd be happy if somebody could review this carefully

[hasnainvirk]

@javier-moreno-tridonic-com It seems that the hard fault issue because of RAM exhaustion with ARMCC is fixed in mbed-o-5.1.3

[TuomoHautamaki]

@javier-moreno-tridonic-com can you close this as the original issue has been solved now. Thank you

[javier-moreno-tridonic-com]

@TuomoHautamaki Well today I tried again a clean checkout of the example using mbed-os 5.1.3 and the fix proposed by @hasnainvirk to save 8KB of RAM but the example still crashes with a HardFault. on the same line.

If you mean that @markus-becker-tridonic-com initial problem is fixed I can close this issue and open a new one about the HardFault and reference this one. Is that what you want?

[hasnainvirk]

@TuomoHautamaki @markus-becker-tridonic-com I had tested it with mbed-OS 5.1.3 and the memory work arounds. It worked fine for me. I just had a Skype session with @javier-moreno-tridonic-com and I demonstrated him the solution. I suspect the OpenOCD tool that Javier is using causes troubles because it uses JTAG. By design Nucleo F401RE uses PB_4(i.e., D5 in terms of Arduino) for JTAG whereas the firefly radio shield is also using D5 for SPI operation. I would suggest a test without using external tools. Use mbed-cli to build and flash like drag and drop. Do not build with debug-info.

[javier-moreno-tridonic-com]

I can finally confirm @hasnainvirk theory. Apparently my debug tools cause some sort of problem that end in a hard fault crash.

@markus-becker-tridonic-com can you close this issue???