NRF52840: BLE Init call hangs our custom firmware application.

[yogeshk19]

Description of defect

We have built a custom board which has a NRF52840 BLE Module from uBlox and is one NINA-B series. The firmware was built based on NRF52840_DK platform as the pinout matches exactly. The firmware application is built using the MBED OS 5.15.5. When the application initializes the BLE Stack using the init call the application firmware hangs. The strange part is the init call hangs after several months after use of the custom board. I am opening this issue as I need help troubleshooting or a way to figure out what about this API call hangs the entire firmware application. Even if I enable mbed tracing, I don't see anything happening after this call to troubleshoot this further. I am looking for guidance as this is affecting our products in the customer location.

logDebug("Initializing BLE Stack...");

//initialize the ble and m_ble is of type Ble
m_ble.init(this, &DeviceManager::OnDeviceManagerInitialized);

Target(s) affected by this defect ?

NRF52840_DK

Toolchain(s) (name and version) displaying this defect ?

NRF52840_DK, nRF52840 SoC

What version of Mbed-os are you using (tag or sha) ?

mbed-os-5.15.5

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli

How is this defect reproduced ?

Run any of the MBED BLE Examples and when the BLE Stack initialization method is called the application hangs.

[mbedmain]

@yogeshk19 thank you for raising this issue.Please take a look at the following comments:

It would help if you could also specify the versions of any tools you are using?

NOTE: If there are fields which are not applicable then please just add 'n/a' or 'None'. This indicates to us that at least all the fields have been considered. Please update the issue header with the missing information.

[pan-]

@yogeshk19 Can you tell us which Bluetooth stack is used ? Cordio or softdevice ?

[yogeshk19]

@pan- The Bluetooth stack used is Cordio. I didn't think the softdevice option was available for 5.15.5 version of the mbed os?

Here is our mbed_app settings file contents.

{
    "target_overrides": {
        "NRF52840_DK": {
            "target.features_add": ["BLE"],
            "target.extra_labels_add": ["CORDIO", "CORDIO_LL", "SOFTDEVICE_NONE", "NORDIC_CORDIO"],
            "target.features_remove": ["CRYPTOCELL310"],
            "target.macros_remove": ["MBEDTLS_CONFIG_HW_SUPPORT"],
            "target.extra_labels_remove": ["SOFTDEVICE_COMMON", "SOFTDEVICE_S140_FULL", "NORDIC_SOFTDEVICE"],
            "platform.stdio-baud-rate": 115200,
            "platform.default-serial-baud-rate": 115200,
            "target.console-uart-flow-control": null,
            "target.bootloader_img" : "..\\mBootloader\\BUILD\\NRF52840_DK\\GCC_ARM-RELEASE\\mBootloader.hex",
            "target.header_offset" : "0x‭‭B000",
            "target.app_offset" : "0x‭‭D000"
        }
    }
}

Thanks, Yogesh

[pan-]

Thanks for the details. If it hangs during the Bluetooth initialization, I think it would hang during the Bluetooth setup.

Can you add traces in this function to understand how far it goes ? The processing should happen in sequence: HCI_OPCODE_RESET then HCI_OPCODE_SET_EVENT_MASK, ...

Are the clock identical between your two boards ?

The initialization can hang if the LF clock source is different:

https://github.com/ARMmbed/mbed-os/blob/mbed-os-5.15.3-rc1/features/FEATURE_BLE/targets/TARGET_NORDIC/TARGET_NORDIC_CORDIO/TARGET_NRF5x/NRFCordioHCIDriver.cpp#L295-L301

[ghost]

What does the custom bootloader do? Does it do FOTA somehow? If so, what stack?

[yogeshk19]

@pan- I have enabled tracing as per the mbed-os tracing documentation for 5.15.5, however I have never been able to get any trace o/p. I am not sure what I am doing wrong. I built the binaries for debug and enabled tracing and yet I don't see any trace o/p except for the debug o/p's i generate in my custom code. I stream the debug thru' the COM ports, since our custom board has a mini USB interface, which we can connect to a computer.

Btw you mentioned enable tracing in this function to see how far it goes? Can you be specific which function are you talking about? Are you saying to enabled tracing in the call to the BLE init method?

Also to answer your earlier questions, the clocks are identical as far as i can say, as our production boards are mass produced. Since I could not get the trace statements, I had added printf statements in the do_initialize() you had mentioned above and nothing is streamed thru' the comm ports in that case either.

Let me know if you have any suggestions as to how best to turn on tracing on mbed os 5.15.5 version.

@jrobeson The custom bootloader allows us to flash a complete custom firmware via the mini USB interface. However we do FOTA over LORA, since the board also has a LORA Module on it. so we can transfer firmware OTA and flash it in the BLE module if any urgent patch needs to be done.

Thanks, Yogesh

[yogeshk19]

@pan- I am trying to get mbed tracing to work, but I can't get it to work. Here is what I am doing to enable tracing for mbed-os 5.15.5. Let me know if I am missing step. I am hoping it doesn't matter if its a release build/debug build, I am guessing the tracing steps are alike.

As shown below I have enabled tracing by adding the "mbed-trace.enable = true" and the trace level setting "mbed-trace.max-level": "TRACE_LEVEL_DEBUG" to the mbed_app.json.

{
    "target_overrides": {
        "NRF52840_DK": {
            "target.features_add": ["BLE"],
            "target.extra_labels_add": ["CORDIO", "CORDIO_LL", "SOFTDEVICE_NONE", "NORDIC_CORDIO"],
            "target.features_remove": ["CRYPTOCELL310"],
            "target.macros_remove": ["MBEDTLS_CONFIG_HW_SUPPORT"],
            "target.extra_labels_remove": ["SOFTDEVICE_COMMON", "SOFTDEVICE_S140_FULL", "NORDIC_SOFTDEVICE"],
            "platform.stdio-baud-rate": 115200,
            "platform.default-serial-baud-rate": 115200,
            "target.console-uart-flow-control": null,
            "target.bootloader_img" : "..\\mBootloader\\BUILD\\NRF52840_DK\\GCC_ARM-RELEASE\\mBootloader.hex",
            "target.header_offset" : "0x‭‭B000",
            "target.app_offset" : "0x‭‭D000",
            "mbed-trace.enable": true,
            "mbed-trace.max-level": "TRACE_LEVEL_DEBUG"
        }
    }
}

I have added the following code in main.cpp.

#include "mbed-trace/mbed_trace.h"
#define TRACE_GROUP  "main"

int main() {
    mbed_trace_init();
    tr_debug("this is debug msg"); //to test if tracing is working

Let me know if I need to add anything else above to successfully enable tracing in my application, so I can add these trace statements in CordioBle.cpp where the BLE Stack is being initialized and so I can atleast provide some information as to where it is currently failing.

Thanks, Yogesh

[yogeshk19]

@pan- Any thoughts?

[yogeshk19]

I was wondering if this issue is getting triaged further so we can make some progress? The reason I am trying to ensure mbed tracing works is so that I can provide mbed team the details it would need to troubleshoot this issue further and also since we can't probe this board, as it is an enclosure and the only way to know what is going on in the board is via the USB o/p. So it would be great if the mbed team to take the time to help with this issue, so we can make some progress here.

Thanks, Yogesh

[nuertey]

Hi @yogeshk19 , Assuming you are compiling with GCC_ARM, to enable the traces, you also have to compile with the debug build profile. The release build profile specifically disables traces. Specifically, the flag "-DNDEBUG" must be removed. Regards Nuertey