Closed JayJaehoonRhee closed 5 years ago
ARM Internal Ref: IOTCELL-2140
Border router has regulatory domain from Korea (9) which is different than the default regulatory domain in the mesh-minimal which is European Union (3). When you match those it has a possibility to work
Thanks, @mikter!
Hi, @JayJaehoonRhee Please change the end node json configuration to use
"regulatory-domain": 9,
(CC: @byungdoochoi )
Thanks for quick replay As you mentioned, I tried to change the regulatory domain 9 to 3, and I can see that there was some change in the logs. But mesh nodes still can not get connection IP address like the ATMEL thread. When I see the logs, it seem to be some problems with TLS. Doesn't mesh-led-control-example work, when I run the Wi-SUN?
[ERR ][tlsl]: Own cert parse eror[0m [ERR ][tlsl]: cert conf fail[0m [ERR ][tlsp]: TLS: library connect fail[0m [DBG ][tlsp]: TLS: finish, eui-64: 6a:02:26:ad:04:0f:18:15[0m [ERR ][eapa]: EAP-TLS: handshake fatal error[0m [INFO][eapa]: EAP-TLS: send FAILURE type TLS id 2 flags ff len 8, eui-64: 6a:02:26:ad:04:0f:18:15[0m [INFO][eapa]: EAP-TLS finish, eui-64: 6a:02:26:ad:04:0f:18:15[0m [DBG ][secl]: prot timeout[0m [DBG ][tlsp]: TLS: finished, eui-64: 6a:02:26:ad:04:0f:18:15 free F[0m
Failure comes from mbed TLS certificate parsing that is made when mbed TLS library initializes on border router:
if (mbedtls_x509_crt_parse(&sec->owncert, cert, cert_len) < 0) {
tr_error("Own cert parse eror");
return -1;
}
It means that own certificate (BR certificate) is configured correctly:
"certificate-header": {
"help": "Certificate header",
"value": "\"wisun_certificates.h\""
},
"own-certificate": {
"help": "Own certificate in PEM format (must be a null terminated c-string)",
"value": "WISUN_SERVER_CERTIFICATE"
},
And found by library, but mbed TLS cannot parse it. This might be due to invalid TLS configuration (no PEM parsing support):
"macros": ["MBEDTLS_USER_CONFIG_FILE=\"source/mbedtls_wisun_config.h\""],
However the above configuration should be ok and include the PEM parse flag:
One reason could be lack of mbed OS heap for mbed TLS to parse the certificate. We have seen that with some platforms. To test that, you could temporary configure nanostack heap smaller:
"heap-size": {
"help": "The amount of static RAM to reserve for nsdynmemlib heap",
"value": 100000
},
e.g. to 60000 and see if it has effect when there should be more memory for mbed OS heap and mbed TLS.
Failure comes from mbed TLS certificate parsing that is made when mbed TLS library initializes on border router:
if (mbedtls_x509_crt_parse(&sec->owncert, cert, cert_len) < 0) { tr_error("Own cert parse eror"); return -1; }
It means that own certificate (BR certificate) is configured correctly:
"certificate-header": { "help": "Certificate header", "value": "\"wisun_certificates.h\"" }, "own-certificate": { "help": "Own certificate in PEM format (must be a null terminated c-string)", "value": "WISUN_SERVER_CERTIFICATE" },
And found by library, but mbed TLS cannot parse it. This might be due to invalid TLS configuration (no PEM parsing support):
"macros": ["MBEDTLS_USER_CONFIG_FILE="source/mbedtls_wisun_config.h""],
However the above configuration should be ok and include the PEM parse flag:
define MBEDTLS_PEM_PARSE_C
One reason could be lack of mbed OS heap for mbed TLS to parse the certificate. We have seen that with some platforms. To test that, you could temporary configure nanostack heap smaller:
"heap-size": { "help": "The amount of static RAM to reserve for nsdynmemlib heap", "value": 100000 },
e.g. to 60000 and see if it has effect when there should be more memory for mbed OS heap and mbed TLS.
Tanks for your reply. I tested with changing nanostack heap from 100000 to 60000 like you mentioned and then the application was started successfully. I could find several handshaking for various processes. But I think the system has one more problem. After the mesh nodes get IP address and mesh application (mesh-led-control-example) is started successfully, when I pressed user button on the F429ZI board, the system hard fault is occurred and system is crashed.
++ MbedOS Fault Handler ++
FaultType: HardFault
Context: R0 : 00000000 R1 : 20010BF8 R2 : 00000000 R3 : 00000000 R4 : 00000001 R5 : 20011610 R6 : 20011590 R7 : 20011598 R8 : 20011514 R9 : 20011511 R10 : 00000000 R11 : 00000000 R12 : 08051D49 SP : 20012668 LR : 0805740B PC : 08057420 xPSR : 410F0000 PSP : 20012600 MSP : 2002FFC0 CPUID: 410FC241 HFSR : 40000000 MMFSR: 00000082 BFSR : 00000000 UFSR : 00000000 DFSR : 00000008 AFSR : 00000000 MMFAR: 00000000 Mode : Thread Priv : Privileged Stack: PSP
-- MbedOS Fault Handler --
++ MbedOS Error Info ++ Error Status: 0x80FF013D Code: 317 Module: 255 Error Message: Fault exception Location: 0x804F927 Error Value: 0x8057420 Current Thread: main Id: 0x20010BF8 Entry: 0x8050BBB StackSize: 0x1000 StackMem: 0x20011748 SP: 0x2002FF58 For more info, visit: https://mbed.com/s/error?error=0x80FF013D&tgt=NUCLEO_F429ZI -- MbedOS Error Info --
= System will be rebooted due to a fatal error = = Reboot count(=1) reached maximum, system will halt after rebooting =^@Start mesh-minimal application
@JayJaehoonRhee
Could you kindly retest with --profile debug
?
The debug build mode will print out the location of failure in detail instead of 0x804F927. Also, please do not discard *.elf file when you are debugging.
This is example of how I build debug mode.
mbed compile -v --profile debug -t GCC_ARM -m K64F
@JayJaehoonRhee Could you kindly retest with
--profile debug
? The debug build mode will print out the location of failure in detail instead of 0x804F927. Also, please do not discard *.elf file when you are debugging. This is example of how I build debug mode.` mbed compile -v --profile debug -t GCC_ARM -m K64F
I did test as debug mode, Nevertheless, there doesn't seem to be a big differences on the logs. Below is cli command I tried.
mbed compile -v --profile debug -t GCC_ARM -m NUCLEO_F429ZI --app-config configs/mesh_wisun_S2LP.json
Thanks
@JayJaehoonRhee Could you kindly run a command similar to following to tell us where it's crashing?
arm-none-eabi-addr2line 0x1ED9F -e ./BUILD/K64F/GCC_ARM-DEBUG/mbed-os-example-ftp.elf
@andrewc-arm I tried like below arm-none-eabi-addr2line 0x804F927 -e ./BUILD/NUCLEO_F429ZI/GCC_ARM-DEBUG/mbed-os-example-mesh-minimal.elf
and the result is below /home/mbed-os-example-mesh-minimal/./mbed-os/features/nanostack/sal-stack-nanostack/source/6LoWPAN/ws/ws_pae_supp.c:875
Thanks.
@JayJaehoonRhee Your point is this one according to you. https://github.com/ARMmbed/mbed-os/blob/master/features/nanostack/sal-stack-nanostack/source/6LoWPAN/ws/ws_pae_supp.c#L875 And it does not make sense. Could you kindly share the exact git tag of your mbed-os?
@andrewc-arm
Actually, I didn't use git clone, I just used "mbed import command" according to github guide.
Below link is github that I referenced. https://github.com/ARMmbed/mbed-os-example-mesh-minimal
[mbed] Working path "/home/company/workspace-mbed/wi-sun/import_test" (directory) [mbed] Program path "/home/company/workspace-mbed/wi-sun/import_test" [mbed] Importing program "mbed-os-example-mesh-minimal" from "https://github.com/ARMmbed/mbed-os-example-mesh-minimal" at latest revision in the current branch [mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #808c45062f94 [mbed] Auto-installing missing Python modules (hidapi, pywin32, wmi)...
There's something different git tag and my source. According to my source code, ws_pae_supp.c:875 likes below.
869 static int8_t ws_pae_supp_parent_eui_64_get(protocol_interface_info_entry_t interface_ptr, uint8_t eui_64) 870 { 871 rpl_dodag_info_t dodag_info; 872 if (!interface_ptr->rpl_domain) { 873 return -1; 874 } 875 struct rpl_instance instance = rpl_control_enumerate_instances(interface_ptr->rpl_domain, NULL); 876 if (instance && rpl_control_read_dodag_info(instance, &dodag_info)) { 877 // Get parent 878 const uint8_t parent_ll_addr = rpl_control_preferred_parent_addr(instance, false); 879 if (parent_ll_addr) { 880 memcpy(eui_64, &parent_ll_addr[8], 8); 881 eui_64[0] |= 0x02; 882 return 0; 883 } 884 } 885 886 return -1; 887 }
Thanks.
I retried test after mbed import from git
<1> mbed import https://github.com/ARMmbed/mbed-os-example-mesh-minimal.git <2> mbed compile -v --profile debug -m NUCLEO_F429ZI -t GCC_ARM --app-config configs/mesh_wisun_S2LP.json I could know the mbed os version supplied from this repository is 5.13.2 . The phenomenon was same. e.i when I press the user button, the system occurred hard fault. So I tried got the debug message, but the result is little bit different. <3> arm-none-eabi-addr2line 0x8090C6B -e ./BUILD/NUCLEO_F429ZI/GCC_ARM-DEBUG/mbed-os-example-mesh-minimal.elf ==> result /home/my/workspace-mbed/wi-sun/clone_source/mbed-os-example-mesh-minimal/./mbed-os/platform/TARGET_CORTEX_M/mbed_fault_handler.c:79 When I found this source from current mbed-os repository, the source tree is little different. I think mbed-os version might be different(5.13.3?).@JayJaehoonRhee The error point you are having is unfortunately incorrect one. We may need a different mbed-os code to pin point the real crash point. I will get back to you within a week or less. If you are in urgency, please enable pyOCD debugging and run it to debug, while pyOCD - gdb is connected, the crash break stop will happen before the mbed_fault_handler which will reveal the true crash point.
@JayJaehoonRhee I created an Mbed-OS PR which will reveal the true crash point. You might want to get the latest mbed-os and apply the PR code to reproduce your failure to get the exact point of crash. Sorry for inconvenience but could you please consider that? After that our core WiSUN engineer can help you.
@andrewc-arm I have a question before I follow your comment. Actually, the source that I've been using is mbed-example-mesh-minimal on mesh sides. When I clone this source, the version of mbedOS I can get is 5.13.2. As I know, the latest version of mbedOS is 5.13.3, then should I replace from 5.13.2 to 5.13.3 from below repository? In other word, should I remove current version of mbedOS from my source tree and then clone the latest version of mbedOS below repository?
MbedOS repository. https://github.com/ARMmbed/mbed-os
There is a known issue in 5.13 that is fixed in 5.13.3 patch and I would suggest you use that for testing and same version should be used for both border router and mesh minimal.
@JayJaehoonRhee
should I replace from 5.13.2 to 5.13.3 from below repository?
Yes. I would change the version like this.
Instead of git checkout master
, you will want to checkout the exact version @mikter mentioned.
cd mbed-os
git fetch
git tag | grep "5.13"
git checkout mbed-os-5.13.3
git checkout -b my_branch
git log
Please make sure that your my_branch
(you can change name), is aligned with mbed-os-5.13.3
from git log
.
Then you apply the code fix of correcting crash point to my_branch
.
git cherry-pick 2c3ce9629eefd3af63657c62a1e40c28f4a1facd
And manually apply the code change of PR
If you find any other difficulties, please let me know.
@andrewc-arm Below is my error message.
++ MbedOS Fault Handler ++
FaultType: HardFault
Context: R0 : 20011694 R1 : 00000000 R2 : 00000000 R3 : 00000000 R4 : 0808C5D5 R5 : 20014250 R6 : 00000000 R7 : 00000000 R8 : 00000000 R9 : 00000000 R10 : 00000000 R11 : 00000000 R12 : 0809529D SP : 20012690 LR : 0800059F PC : 080004B8 xPSR : 410F0000 PSP : 20012628 MSP : 2002FFB0 CPUID: 410FC241 HFSR : 40000000 MMFSR: 00000082 BFSR : 00000000 UFSR : 00000000 DFSR : 00000008 AFSR : 00000000 MMFAR: 00000000 Mode : Thread Priv : Privileged Stack: PSP
-- MbedOS Fault Handler --
++ MbedOS Error Info ++ Error Status: 0x80FF013D Code: 317 Module: 255 Error Message: Fault exception Location: 0x746C Error Value: 0x80B9710 Current Thread: main Id: 0x20010C90 Entry: 0x8093017 StackSize: 0x1000 StackMem: 0x20011810 SP: 0x67616E61 For more info, visit: https://mbed.com/s/error?error=0x80FF013D&tgt=NUCLEO_F429ZI -- MbedOS Error Info --
= System will be rebooted due to a fatal error = = Reboot count(=1) reached maximum, system will halt after rebooting =^@
When I execute "arm-none-eabi-addr2line", I don't know witch one is address. Location or Entry? When I use location as address, the message is blow ??:0 And when I use entry as address , the message like below /home/xxx/workspace-mbed/wi-sun/mbed-os-example-mesh-minimal/./mbed-os/rtos/TARGET_CORTEX/mbed_boot.c:86
Which one is right? or anything else?
Why my source tree differ from source tree at "https://github.com/ARMmbed/mbed-os" even though I did "git fetch"
@JayJaehoonRhee
Could you please addr2line
following information?
LR : 0800059F
PC : 080004B8
PC is for current failure and LR is for the caller of the failure. Maybe you applied the PR wrong, but still this is useful. Please let us know.
@andrewc-arm LR /home/xxx/workspace-mbed/wi-sun/mbed-os-example-mesh-minimal/./mbed-os/drivers/DigitalOut.h:80
PC /home/xxx/workspace-mbed/wi-sun/mbed-os-example-mesh-minimal/./mbed-os/targets/TARGET_STM/gpio_object.h:64
OK. Then this is the callback.
It sounds like it was merely trying to write DigitalOut but either obj
was NULL or corrupted.
*obj->reg_clr = obj->mask << 16;
Did you change any code near GPIO or DigitalOut? Or perhaps assigned wrong GPIO on mbed_app.json?
@JayJaehoonRhee
I see that from your configs_20190821.zip following parts especially around LED
.
mesh_wisun_S2LP.json
"config": {
"enable-led-control-example": true,
"LED": "NC",
"BUTTON": "NC",
"RELAY_CONTROL": "NC",
"BUTTON_MODE": "PullUp"
},
"target_overrides": {
"NUCLEO_F429ZI": {
"LED": "LED_RED",
"BUTTON": "USER_BUTTON",
"BUTTON_MODE": "PullDown"
}
Could you make sure you are building with NUCLEO_F429ZI
+ this config file or otherwise please add rules like NUCLEO_F429ZI
? If you are building with other not listed target, you will get LED
== NC
(not connected).
Wisun_Stm_s2lp_RF.json
"LED": "NC",
If this device is going to blink, it will crash like above. Could you please add LED pin information there?
@andrewc-arm There is no any code change by me near GPIO and DigitalOut. I just changed d11_configuration from PA_7 to PB_5 for NUCLEO_F429ZI on target.json .
And I just run the compile with options like below
mbed compile -m NUCLEO_F429ZI -t GCC_ARM --app-config configs/mesh_wisun_S2LP.json
and my config files is already same with your comment.
@andrewc-arm I fixed this issue. The cause was in mesh-led-control-example.cpp code. DigitalOut output(MBED_CONF_APP_RELAY_CONTROL, 1);
In the case of NUCLEO_F429ZI, there is no pin definition of relay control. So I remarked the codes related relay control.
Thanks.
This issue was fixed in https://github.com/ARMmbed/mbed-os-example-mesh-minimal/pull/283
Below error comes in mbed-os-example-mesh-minimal when my Router try to connect Boarder router. i am using wisun_certificates.h file. error is comming from SANS verification function in tls.
[DBG ][tlsl]: Ignored subject alt name: 1
[ERR ][tlsl]: invalid cert
please help to solve above issue.
Thanks
The certificates on wisun_certificates.h does not have all wisun specific fields. This is just warning about that:
[ERR ][tlsl]: invalid cert
It does not affect the authentication/connecting.
in function tls_sec_prot_lib_subject_alternative_name_validate it return error and after this error EAPOL handshaking fail message display.
Could you provide the full log.
Error happens when BR receives client hello:
[2K[39m[INFO][eapa]: EAP-TLS: recv RESPONSE type TLS id 2 flags 0 len 98, eui-64 5a:87:31:56:c8:66:55:b1[0m
[2K[90m[DBG ][tlsp]: TLS: start, eui-64: 5a:87:31:56:c8:66:55:b1[0m
[2K[90m[DBG ][tlsp]: TLS QUEUE add index: 0, eui-64: 5a:87:31:56:c8:66:55:b1[0m
[2K[31m[ERR ][tlsp]: TLS: error, eui-64: 5a:87:31:56:c8:66:55:b1[0m
[2K[90m[DBG ][tlsp]: TLS: finish, eui-64: 5a:87:31:56:c8:66:55:b1[0m
For some reason TLS returns error. Might be something in library or its adaptation. You could enable extra mbed tls traces by enabling TLS_SEC_PROT_LIB_TLS_DEBUG:
/mbed-os/features/nanostack/sal-stack-nanostack/source/Security/protocols/tls_sec_prot/tls_sec_prot_lib.c
//#define TLS_SEC_PROT_LIB_TLS_DEBUG // Enable mbed TLS debug traces
and take a new trace from BR.
I am using NUCLEO-F429ZI as Router and BorderRouter. R_LOG.log BR_LOG.log
Might be that for some reason nanostack memory is exhausted. Handling fails to ssl send:
[2K[90m[DBG ][tlsl]: 4 .\mbed-os\features\mbedtls\src\ssl_tls.c 4534 0000: 14 03 03 00 01 01 ...... [0m
[2K[90m[DBG ][tlsl]: 2 .\mbed-os\features\mbedtls\src\ssl_tls.c 3828 => flush output [0m
[2K[90m[DBG ][tlsl]: 2 .\mbed-os\features\mbedtls\src\ssl_tls.c 3847 message length: 6, out_left: 6 [0m
Which calls: -> tls_sec_prot_lib_ssl_send -> tls_sec_prot_tls_send()
and most probably returns from here: if (!data->tls_send.data) { return -1; }
Memory is configured by:
"mbed-mesh-api.heap-size": 65535,
Are you using default BR configuration?
this is my BR configuration json file
Ok, please update to latest configuration file (and BR version):
https://github.com/ARMmbed/nanostack-border-router/blob/master/configs/Wisun_Stm_s2lp_RF.json
This configuration is obsolete:
"heap-size": {
"help": "The amount of static RAM to reserve for nsdynmemlib heap",
"value": 60000
},
and this should be used instead:
"mbed-mesh-api.heap-size": 65535,
As in the example .json. The example .json should work as it is with the latest BR software.
Now you run out of mbed os heap:
[2K[90m[DBG ][tlsl]: 3 .\mbed-os\features\mbedtls\src\sslsrv.c 3339 0010: 9c b9 fc 95 5f 2c 1e ec 47 3f f6 c8 99 5c 9f a5 ....,..G?..... [0m
[2K[90m[DBG ][tlsl]: 1 .\mbed-os\features\mbedtls\src\ssl_srv.c 3412 mbedtls_pk_sign() returned -16 (-0x0010)
Maybe "mbed-mesh-api.heap-size": 50000, would work.
For some reason trace indidates that heap size would be 126926:
[2K[39m[INFO][app ]: Heap size: 126926, Reserved: 25132, Reserved max: 57640, Alloc fail: 0[0m
@artokin was there some configuraton issues with the br heap?
nanostack-border-router is using external heap, please see https://github.com/ARMmbed/nanostack-border-router/blob/master/source/nanostack_heap_region.c#L15
External heap usage is controlled by: https://github.com/ARMmbed/nanostack-border-router/blob/master/configs/Wisun_Stm_s2lp_RF.json#L23
so what size i have to set for heap for NUCLEO-F429ZI ? what should be configuration ??
BR_LOG.log R_LOG.log BR_mbed_app.txt
Now i change in BR configuration file and get Handshake success message but still my router is not connected.
Router and Border router connected successfully. Help size adjustment problem only.
Thanks mikaleppanen and artokin.
I have 3 set of NUCLEO_F429ZI & X-NUCLEO-S2868A1 and I tried Wi-SUN test using mbed-os-example-mesh-minimal and nanostack-border-router. 1 set of them was used border router and others were downloaded as mesh minimal. First of all, I fixed NUCLEO_F429ZI and X-NUCLEO-S2868A1 according to below links I mean, in the case of NUCLEO_F429ZI, I re moved the SB121 register and then mounted at SB122 it. Also I removed R11 and mounted R6 the register on the X-NUCLEO-S2868A1. Because these borders conflict SPI and Ethernet. And I changed the codes "d11_configuration : PB_5" in the /mbed-os/targets/target.json ============ links ============= https://os.mbed.com/teams/ST/wiki/Nucleo-144pins-ethernet-spi-conflict https://github.com/ARMmbed/stm-spirit1-rf-driver
mbed-os-mesh-minimal codes
"NUCLEO_F429ZI": { "inherits": ["FAMILY_STM32"], "supported_form_factors": ["ARDUINO"], "core": "Cortex-M4F", "config": { "d11_configuration": { "help": "Value: PA_7 for the default board configuration, PB_5 in case of solder bridge (SB121 off/ SB122 on)", "value": "PB_5", "macro_name": "STM32_D11_SPI_ETHERNET_PIN" }, "clock_source": { "help": "Mask value : USE_PLL_HSE_EXTC | USE_PLL_HSE_XTAL (need HW ) | USE_PLL_HSI", "value": "USE_PLL_HSE_EXTC|USE_PLL_HSI", "macro_name": "CLOCK_SOURCE" } },
nanostack-border-routerNUCLEO_F429ZI": { "inherits": ["FAMILY_STM32"], "supported_form_factors": ["ARDUINO"], "core": "Cortex-M4F", "config": { "d11_configuration": { "help": "Value: PA_7 for the default board configuration, PB_5 in case of solder bridge (SB121 off/ SB122 on)", "value": "PB_5", "macro_name": "STM32_D11_SPI_ETHERNET_PIN" }, "clock_source": { "help": "Mask value : USE_PLL_HSE_EXTC | USE_PLL_HSE_XTAL (need HW ) | USE_PLL_HSI", "value": "USE_PLL_HSE_EXTC|USE_PLL_HSI", "macro_name": "CLOCK_SOURCE" } },
I attatch my log and Wi-SUN config files. Please take a look at theses. Thanks
2019-08-19-103419-ACM0-WiSUN_BR.txt 2019-08-19-103533-ACM1-WiSUN_Node1.txt 2019-08-19-103604-ACM2-WiSUN_Node2.txt
configs.zip