Device Management tests improvements

screamerbg commented 5 years ago

This PR aims to improve the following aspects of Device Management tests for SPDMC:

Use wait_ms() instead of while() loop, which was previously used to prevent strange behavior with some Targets in TICKLESS mode.
Speed-up Pelion DM registration and PDM device directory test cases by querying on every 100ms instead of previously hardcoded 5 seconds wait. Also increase the timeout for these test cases to 10 seconds.
Prevent boot loop due to Mbed OS crash by limiting the times the host test python script will respond (currently up to 5 {{device_ready}} messages)
Improve the output of both Connect and Update test suites for better readability when using verbose mode mbed test ... -v. See example 1 below.
Correctly report the number of successful test runs for a routine if ran more than once 9, e.g. "Initialize BD+DS", "Connect to Network" etc occur more than once in a test run. See example 2.
Reduce the scope of networking tests over Cellular due to tests timing out caused by low Cellular speeds. The tests will be restored once dynamic test timeouts are relased with greentea/htrun. PR for the dynamic test timeout feature - https://github.com/ARMmbed/htrun/pull/228

Example 1

[1543713891.05][CONN][RXD] >>> Running case #1: 'Initialize QSPIF+LFS'...
[1543713891.05][CONN][INF] found KV pair in stream: {{__testcase_name;Post-update Identity}}, queued...
[1543713891.05][SERI][TXD] {{iteration;0}}
[1543713891.06][CONN][INF] found KV pair in stream: {{__testcase_start;Initialize QSPIF+LFS}}, queued...
[1543713891.08][CONN][RXD] [INFO] Attempting to initialize storage.
[1543713891.11][CONN][RXD] >>> 'Initialize QSPIF+LFS': 1 passed, 0 failed
[1543713891.11][CONN][INF] found KV pair in stream: {{__testcase_finish;Initialize QSPIF+LFS;1;0}}, queued...
[1543713891.12][CONN][RXD]
[1543713891.12][CONN][RXD] >>> Running case #2: 'Connect to WiFi'...
[1543713891.14][CONN][RXD] [INFO] Attempting to connect to network.
[1543713891.14][CONN][INF] found KV pair in stream: {{__testcase_start;Connect to WiFi}}, queued...
[1543713894.89][CONN][RXD] [INFO] Connected to network successfully. IP address: 192.168.101.64
[1543713894.90][CONN][INF] found KV pair in stream: {{__testcase_finish;Connect to WiFi;1;0}}, queued...
[1543713894.92][CONN][RXD] >>> 'Connect to WiFi': 1 passed, 0 failed
[1543713894.93][CONN][RXD]
[1543713894.93][CONN][RXD] >>> Running case #3: 'Format LFS'...
[1543713894.93][CONN][INF] found KV pair in stream: {{__testcase_start;Format LFS}}, queued...
[1543713894.95][CONN][RXD] [INFO] Resetting storage to a clean state for test.
[1543713895.15][CONN][RXD] [INFO] Storage format successful.
[1543713895.18][CONN][RXD] >>> 'Format LFS': 1 passed, 0 failed
[1543713895.18][CONN][RXD]
[1543713895.18][CONN][INF] found KV pair in stream: {{__testcase_finish;Format LFS;1;0}}, queued...
[1543713895.20][CONN][RXD] >>> Running case #4: 'Initialize Simple PDMC'...
[1543713895.20][CONN][INF] found KV pair in stream: {{__testcase_start;Initialize Simple PDMC}}, queued...
[1543713908.20][CONN][RXD] [INFO] Simple PDMC initialization successful.
[1543713908.22][CONN][INF] found KV pair in stream: {{__testcase_finish;Initialize Simple PDMC;1;0}}, queued...
[1543713908.23][CONN][RXD] >>> 'Initialize Simple PDMC': 1 passed, 0 failed

Example 2

[1543714029.13][CONN][RXD] >>> Running case #11: 'Post-update Erase'...
[1543714029.13][CONN][RXD] >>> 'Post-update Erase': 1 passed, 0 failed
[1543714029.13][CONN][INF] found KV pair in stream: {{__testcase_start;Post-update Erase}}, queued...
[1543714029.13][CONN][INF] found KV pair in stream: {{__testcase_finish;Post-update Erase;1;0}}, queued...
[1543714029.15][CONN][RXD]
[1543714029.15][CONN][RXD] >>> Running case #2: 'Connect to WiFi'...
[1543714029.15][CONN][INF] found KV pair in stream: {{__testcase_start;Connect to WiFi}}, queued...
[1543714029.16][CONN][RXD] [INFO] Attempting to connect to network.
[1543714032.90][CONN][RXD] [INFO] Connected to network successfully. IP address: 192.168.101.64
[1543714032.90][CONN][INF] found KV pair in stream: {{__testcase_finish;Connect to WiFi;2;0}}, queued...
[1543714032.91][CONN][RXD] >>> 'Connect to WiFi': 2 passed, 0 failed
[1543714032.93][CONN][RXD]
[1543714032.93][CONN][RXD] >>> Running case #4: 'Initialize Simple PDMC'...
[1543714032.93][CONN][INF] found KV pair in stream: {{__testcase_start;Initialize Simple PDMC}}, queued...
[1543714035.71][CONN][RXD] [INFO] Simple PDMC initialization successful.
[1543714035.73][CONN][INF] found KV pair in stream: {{__testcase_finish;Initialize Simple PDMC;2;0}}, queued...
[1543714035.74][CONN][RXD] >>> 'Initialize Simple PDMC': 2 passed, 0 failed

+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+--------+--------------------+
| target                  | platform_name       | test suite                                         | test case                               | passed | failed | result | elapsed_time (sec) |
+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+--------+--------------------+
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Connect to WiFi                         | 2      | 0      | OK     | 3.75               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Format LFS                              | 1      | 0      | OK     | 0.25               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize QSPIF+LFS                    | 2      | 0      | OK     | 0.06               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize Simple PDMC                  | 2      | 0      | OK     | 2.8                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Bootstrap & Reg.                 | 1      | 0      | OK     | 13.8               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Directory                        | 1      | 0      | OK     | 0.86               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Download                | 1      | 0      | OK     | 90.09              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Prepare                 | 1      | 0      | OK     | 0.59               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Update                  | 1      | 0      | OK     | 15.27              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Re-register                      | 1      | 0      | OK     | 5.7                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Erase                       | 1      | 0      | OK     | 0.0                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Identity                    | 1      | 0      | OK     | 0.11               |
+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+--------+--------------------+

screamerbg commented 5 years ago

Test results for this PR. Note that the Nuvoton failures are due to the specific Nuvoton SD driver. The Nuvoton team has been notified about these issues.

2018-12-02 15:02:30 TEST RUNNER RESULTS:
+-------------------------+---------------------+----------------------------------------------------+---------+---------+
| target                  | platform_name       | test suite                                         | result  |    time |
+-------------------------+---------------------+----------------------------------------------------+---------+---------+
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   60.97 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-basic-stress-net-fs | TIMEOUT |  635.23 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   44.09 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-basic-fs-threaded   | TIMEOUT |  633.41 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |    65.1 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-basic-fs-single     | TIMEOUT |  332.69 |
| NUMAKER_PFM_M487-ARM    | NUMAKER_PFM_M487    | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  133.57 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   54.62 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |  105.83 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   44.16 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |  119.43 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |   88.01 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |   170.3 |
| K66F-ARM                | K66F                | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |   169.8 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-basic-net-single    | OK      |    44.4 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |  117.07 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   34.15 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |  150.76 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |   77.93 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |  217.91 |
| NUCLEO_F429ZI-ARM       | NUCLEO_F429ZI       | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  166.41 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   83.75 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |  100.88 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   71.14 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |   79.64 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  126.94 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |  104.81 |
| MTB_USI_WM_BN_BM_22-ARM | MTB_USI_WM_BN_BM_22 | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  320.11 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   55.42 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-basic-stress-net-fs | TIMEOUT |  628.32 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   41.17 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-basic-fs-threaded   | TIMEOUT |  280.15 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  141.63 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-basic-fs-single     | TIMEOUT |  326.16 |
| NUMAKER_PFM_NUC472-ARM  | NUMAKER_PFM_NUC472  | simple-mbed-cloud-client-tests-dev_mgmt-update     | TIMEOUT | 1279.93 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   54.64 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |   62.48 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   42.45 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |    51.8 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  103.87 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |   71.84 |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  187.52 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-basic-net-single    | OK      |  114.19 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |   167.0 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   97.27 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |  149.32 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  158.15 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |  217.27 |
| UBLOX_EVK_ODIN_W2-ARM   | UBLOX_EVK_ODIN_W2   | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  471.03 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-basic-net-single    | OK      |    46.4 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |   56.83 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   35.53 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |   49.17 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |    93.8 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |   67.45 |
| DISCO_F413ZH-ARM        | DISCO_F413ZH        | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  177.54 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-basic-net-single    | OK      |    42.5 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |   98.95 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   31.85 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |  122.26 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |   74.23 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |  178.05 |
| NUCLEO_F746ZG-ARM       | NUCLEO_F746ZG       | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  154.02 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-basic-net-single    | FAIL    |   49.62 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-basic-stress-net-fs | FAIL    |   45.18 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-basic-net-threaded  | FAIL    |   45.15 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-basic-fs-threaded   | TIMEOUT |  621.63 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  102.46 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-basic-fs-single     | TIMEOUT |  321.32 |
| NUMAKER_IOT_M487-ARM    | NUMAKER_IOT_M487    | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |   215.3 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-basic-net-single    | OK      |  248.77 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |  248.78 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |  253.52 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |  146.56 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  153.46 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |  213.36 |
| UBLOX_C030_U201-ARM     | UBLOX_C030_U201     | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  1062.8 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   56.65 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |  112.58 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |   43.53 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |  131.37 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |   91.81 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |  189.44 |
| K64F-ARM                | K64F                | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  176.78 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-basic-net-single    | OK      |   77.11 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-basic-stress-net-fs | OK      |   96.05 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-basic-net-threaded  | OK      |    63.9 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-basic-fs-threaded   | OK      |   80.55 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-dev_mgmt-connect    | OK      |  122.79 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-basic-fs-single     | OK      |   106.2 |
| MTB_MXCHIP_EMW3166-ARM  | MTB_MXCHIP_EMW3166  | simple-mbed-cloud-client-tests-dev_mgmt-update     | OK      |  250.49 |
+-------------------------+---------------------+----------------------------------------------------+---------+---------+

CC @samchuarm

MarceloSalazar commented 5 years ago

This requires more testing and investigation.

Initial test shows failures in net-threaded:

+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+---------+--------------------+
| target                  | platform_name       | test suite                                         | test case                               | passed | failed | result  | elapsed_time (sec) |
+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+---------+--------------------+
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff     1            | 1      | 0      | OK      | 9.34               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff     4            | 1      | 0      | OK      | 6.63               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff    16            | 1      | 0      | OK      | 5.93               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff    64            | 1      | 0      | OK      | 5.77               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff   256            | 1      | 0      | OK      | 5.69               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff  1024            | 1      | 0      | OK      | 5.45               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff  4096            | 1      | 0      | OK      | 4.86               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff 16384            | 1      | 0      | OK      | 4.71               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS format                        | 1      | 0      | OK      | 0.2                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | QSPIF+LFS 2 files, buff 4b/256b         | 1      | 0      | OK      | 12.12              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | QSPIF+LFS 3 files, buff 256b/1kb/4kb    | 1      | 0      | OK      | 16.07              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | QSPIF+LFS format                        | 1      | 0      | OK      | 0.18               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi   128 buffer                       | 1      | 0      | OK      | 11.01              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi   256 buffer                       | 1      | 0      | OK      | 6.55               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi  1024 buffer                       | 1      | 0      | OK      | 3.92               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi  4096 buffer                       | 1      | 0      | OK      | 3.1                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi network setup                      | 1      | 0      | OK      | 9.9                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | WiFi 1 thread                           | 0      | 0      | ERROR   | 0.0                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | WiFi 2 threads                          | 0      | 0      | SKIPPED | 0.0                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | WiFi network setup                      | 1      | 0      | OK      | 0.0                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS format                        | 1      | 0      | OK      | 0.2                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS+WiFi 1 thread, dl, file seq.  | 1      | 0      | OK      | 6.77               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS+WiFi 2 threads, dl, 1kb       | 1      | 0      | OK      | 9.99               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS+WiFi 3 threads, dl, 256b, 1kb | 1      | 0      | OK      | 15.9               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | WiFi network setup                      | 1      | 0      | OK      | 3.58               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Connect to WiFi                         | 2      | 0      | OK      | 3.57               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Format LFS                              | 1      | 0      | OK      | 0.25               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Initialize QSPIF+LFS                    | 2      | 0      | OK      | 0.03               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Initialize Simple PDMC                  | 2      | 0      | OK      | 2.83               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion Bootstrap & Reg.                 | 1      | 0      | OK      | 14.51              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion Directory                        | 1      | 0      | OK      | 0.98               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion Re-register                      | 1      | 0      | OK      | 8.56               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Post-reset Identity                     | 1      | 0      | OK      | 0.11               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M GET                      | 1      | 0      | OK      | 0.86               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M POST                     | 1      | 0      | OK      | 0.43               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M PUT                      | 1      | 0      | OK      | 9.47               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M SET                      | 1      | 0      | OK      | 0.44               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Connect to WiFi                         | 2      | 0      | OK      | 5.16               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Format LFS                              | 1      | 0      | OK      | 0.23               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize QSPIF+LFS                    | 2      | 0      | OK      | 0.04               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize Simple PDMC                  | 2      | 0      | OK      | 2.79               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Bootstrap & Reg.                 | 1      | 0      | OK      | 14.36              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Directory                        | 1      | 0      | OK      | 1.07               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Download                | 1      | 0      | OK      | 89.27              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Prepare                 | 1      | 0      | OK      | 0.58               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Update                  | 1      | 0      | OK      | 15.17              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Re-register                      | 1      | 0      | OK      | 6.11               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Erase                       | 1      | 0      | OK      | 0.0                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Identity                    | 1      | 0      | OK      | 0.13               |
+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+---------+--------------------+

Re-running this net-threaded shows hardfault:

[1543856762.77][CONN][RXD] [BOOT] SHA256: AA670AFAC16724F31E68783ACF0CF83BD1EBE0F2E85DE3ECA4DECE4821D9BA91
[1543856762.77][CONN][RXD] [BOOT] Version: 1543856732
[1543856762.77][CONN][RXD] [BOOT] Slot 0 is empty
[1543856762.77][CONN][RXD] [BOOT] Active firmware up-to-date
[1543856762.77][CONN][RXD] [BOOT] Application's start address: 0x8010400
[1543856762.77][CONN][RXD] [BOOT] Application's jump address: 0x8011041
[1543856762.77][CONN][RXD] [BOOT] Application's stack address: 0x20018000
[1543856762.77][CONN][RXD] [BOOT] Forwarding to application...
[1543856762.77][CONN][RXD] {Lmbedmbedmbedmbedmbedmbedmbedmbed
[1543856762.77][CONN][RXD] >>> Running 3 test cases...
[1543856762.77][CONN][INF] found SYNC in stream: {{__sync;549d16d8-9345-47f3-8b77-3d064c9314be}} it is #0 sent, queued...
[1543856762.77][CONN][INF] found KV pair in stream: {{__version;1.3.0}}, queued...
[1543856762.77][CONN][INF] found KV pair in stream: {{__timeout;600}}, queued...
[1543856762.77][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued...
[1543856762.77][CONN][INF] found KV pair in stream: {{__testcase_count;3}}, queued...
[1543856762.79][HTST][INF] sync KV found, uuid=549d16d8-9345-47f3-8b77-3d064c9314be, timestamp=1543856762.774000
[1543856762.79][HTST][INF] DUT greentea-client version: 1.3.0
[1543856762.79][HTST][INF] setting timeout to: 600 sec
[1543856762.79][HTST][INF] host test class: '<class 'mbed_host_tests.host_tests.default_auto.DefaultAuto'>'
[1543856762.79][HTST][INF] host test setup() call...
[1543856762.79][HTST][INF] CALLBACKs updated
[1543856762.79][HTST][INF] host test detected: default_auto
[1543856762.81][CONN][RXD]
[1543856762.81][CONN][RXD] >>> Running case #1: 'WiFi network setup'...
[1543856762.81][CONN][INF] found KV pair in stream: {{__testcase_name;WiFi network setup}}, queued...
[1543856762.81][CONN][INF] found KV pair in stream: {{__testcase_name;WiFi 1 thread}}, queued...
[1543856762.81][CONN][INF] found KV pair in stream: {{__testcase_name;WiFi 2 threads}}, queued...
[1543856762.81][CONN][INF] found KV pair in stream: {{__testcase_start;WiFi network setup}}, queued...
[1543856763.32][CONN][RXD]
[1543856763.32][CONN][RXD]
[1543856763.32][CONN][RXD] ++ MbedOS Error Info ++
[1543856763.32][CONN][RXD] Error Status: 0x80FF0100 Code: 256 Module: 255
[1543856763.32][CONN][RXD] Error Message: Fatal Run-time error
[1543856763.32][CONN][RXD] Location: 0x801B1E7
[1543856763.32][CONN][RXD] Error Value: 0x0
[1543856763.34][CONN][RXD] Current Thread: Id: 0x20002C20 Entry: 0x80224C9 StackSize: 0x1770 StackMem: 0x200014B0 SP: 0x20002A80
[1543856763.34][CONN][RXD] For more info, visit: https://armmbed.github.io/mbedos-error/?error=0x80FF0100
[1543856763.34][CONN][RXD] -- MbedOS Error Info --
[1543856763.34][CONN][RXD] ISM43362Interface: ERROR cannot read firmware version

screamerbg commented 5 years ago

@MarceloSalazar I'm not seeing these issues neither in the testing system, neither on my setup. But we should definitely investigate further. Perhaps your board firmware is old - see the message from the driver. You can update the WiFi module firmware from this page - https://www.st.com/en/evaluation-tools/b-l475e-iot01a.html

@MarceloSalazar note that this PR doesn't really touch on the networking tests and therefore the issues you reported shouldn't gate this PR. Also, looking at your test results, the device management tests are working fine.

MarceloSalazar commented 5 years ago

Re-tested. Issues are not related to changes on this PR.

PelionIoT / simple-mbed-cloud-client

Device Management tests improvements #57