Feature: Basic tests + Pelion DM test improvements

screamerbg commented 5 years ago

The PR aims to improve some of the existing Pelion DM tests for stability and also introduces basic tests to validate the fundamental platform underneath PDMC:

Filesystem tests

single-threaded test with write buffer 1 byte, 4, 16, 64, 256, 1kb, 4kb, 16kb, 64kb
multi-threaded test with write buffer 1 byte, 4, 16, 64, 256, 1kb, 4kb, 16kb, 64kb

Network/Download tests

single-threaded test with receive buffer sizes 128 bytes, 256, 1 kb, 2kb, 4kb
multi-threaded test with receive buffer size 1kb and up to 3 simultaneous threads/downloads

Network & Filesystem stress tests

1 thread (sequential) - 1 download (1kb buff), 1 file thread (1kb buff)
2 parallel threads - 1 download, 1 file thread (1kb buff)
3 parallel threads - 1 download, 2 file (256 bytes, 1 kb buff)

Here's an example output:

$ mbed test -t ARM -m DISCO_L475VG_IOT01A -n simple-mbed-cloud-client-tests-basics* -v
[SNIP]
+-------------------------+---------------------+-----------------------------------------------------+------------------------------------------+--------+--------+--------+--------------------+
| target                  | platform_name       | test suite                                          | test case                                | passed | failed | result | elapsed_time (sec) |
+-------------------------+---------------------+-----------------------------------------------------+------------------------------------------+--------+--------+--------+--------------------+
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff     1                | 1      | 0      | OK     | 17.89              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff     4                | 1      | 0      | OK     | 15.19              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff    16                | 1      | 0      | OK     | 14.47              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff    64                | 1      | 0      | OK     | 14.24              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff   256                | 1      | 0      | OK     | 14.24              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff  1024                | 1      | 0      | OK     | 12.87              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff  4096                | 1      | 0      | OK     | 8.72               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff 16384                | 1      | 0      | OK     | 6.34               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff 65536                | 1      | 0      | OK     | 5.64               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT format                            | 1      | 0      | OK     | 0.34               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 2 files, buff 1b/256b             | 1      | 0      | OK     | 32.01              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 3 files, buff 256b/1kb/4kb        | 1      | 0      | OK     | 38.64              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 4 files, buff 1b/256b/1kb/4kb     | 1      | 0      | OK     | 57.39              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT format                            | 1      | 0      | OK     | 0.33               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi   256 buffer                        | 1      | 0      | OK     | 6.44               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  1k buffer                          | 1      | 0      | OK     | 3.91               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  2k buffer                          | 1      | 0      | OK     | 3.2                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  4k buffer                          | 1      | 0      | OK     | 3.2                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  8k buffer                          | 1      | 0      | OK     | 3.2                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi network setup                       | 1      | 0      | OK     | 3.83               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 1 thread                            | 1      | 0      | OK     | 4.07               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 2 threads                           | 1      | 0      | OK     | 8.91               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 3 threads                           | 1      | 0      | OK     | 13.45              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 4 threads                           | 1      | 0      | OK     | 24.88              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi network setup                       | 1      | 0      | OK     | 5.28               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT format                            | 1      | 0      | OK     | 0.41               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 1 thread, dl, file seq.      | 1      | 0      | OK     | 3.93               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 2 threads, dl, 1kb           | 1      | 0      | OK     | 16.72              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 3 threads, dl, 1b, 1kb       | 1      | 0      | OK     | 35.31              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 4 threads, dl, 1b, 256b, 1kb | 1      | 0      | OK     | 50.75              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | WiFi network setup                       | 1      | 0      | OK     | 3.81               |
+-------------------------+---------------------+-----------------------------------------------------+------------------------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 31 OK
mbedgt: completed in 570.03 sec

Based on https://github.com/ARMmbed/mbed-stress-test by @marcuschangarm.

screamerbg commented 5 years ago

Here's an example report for all SMCC tests, including the ones from this PR:

+-------------------------+---------------------+-----------------------------------------------------+------------------------------------------+--------+--------+--------+--------------------+
| target                  | platform_name       | test suite                                          | test case                                | passed | failed | result | elapsed_time (sec) |
+-------------------------+---------------------+-----------------------------------------------------+------------------------------------------+--------+--------+--------+--------------------+
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff     1                | 1      | 0      | OK     | 17.94              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff     4                | 1      | 0      | OK     | 15.18              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff    16                | 1      | 0      | OK     | 14.4               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff    64                | 1      | 0      | OK     | 14.31              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff   256                | 1      | 0      | OK     | 14.2               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff  1024                | 1      | 0      | OK     | 12.94              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff  4096                | 1      | 0      | OK     | 8.67               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff 16384                | 1      | 0      | OK     | 6.34               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff 65536                | 1      | 0      | OK     | 5.72               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT format                            | 1      | 0      | OK     | 0.34               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 2 files, buff 1b/256b             | 1      | 0      | OK     | 32.03              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 3 files, buff 256b/1kb/4kb        | 1      | 0      | OK     | 38.64              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 4 files, buff 1b/256b/1kb/4kb     | 1      | 0      | OK     | 57.44              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT format                            | 1      | 0      | OK     | 0.33               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi   256 buffer                        | 1      | 0      | OK     | 6.49               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  1k buffer                          | 1      | 0      | OK     | 3.92               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  2k buffer                          | 1      | 0      | OK     | 3.27               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  4k buffer                          | 1      | 0      | OK     | 3.25               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  8k buffer                          | 1      | 0      | OK     | 3.56               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-single    | WiFi network setup                       | 1      | 0      | OK     | 4.75               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 1 thread                            | 1      | 0      | OK     | 4.03               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 2 threads                           | 1      | 0      | OK     | 8.8                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 3 threads                           | 1      | 0      | OK     | 20.48              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 4 threads                           | 1      | 0      | OK     | 24.44              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi network setup                       | 1      | 0      | OK     | 3.75               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT format                            | 1      | 0      | OK     | 0.33               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 1 thread, dl, file seq.      | 1      | 0      | OK     | 3.93               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 2 threads, dl, 1kb           | 1      | 0      | OK     | 16.76              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 3 threads, dl, 1b, 1kb       | 1      | 0      | OK     | 35.31              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 4 threads, dl, 1b, 256b, 1kb | 1      | 0      | OK     | 50.65              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basics-stress-net-fs | WiFi network setup                       | 1      | 0      | OK     | 5.76               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Connect to WiFi                          | 1      | 0      | OK     | 7.66               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Format FAT                               | 1      | 0      | OK     | 0.35               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Initialize SD+FAT                        | 1      | 0      | OK     | 0.05               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Initialize Simple PDMC                   | 1      | 0      | OK     | 3.73               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Pelion DM Bootstrap & Reg.               | 1      | 0      | OK     | 16.0               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Pelion DM Directory                      | 1      | 0      | OK     | 6.5                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Pelion DM Re-register                    | 1      | 0      | OK     | 6.97               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Post-reset Identity                      | 1      | 0      | OK     | 2.47               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Resource LwM2M GET                       | 1      | 0      | OK     | 0.86               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Resource LwM2M POST                      | 1      | 0      | OK     | 0.69               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Resource LwM2M PUT                       | 1      | 0      | OK     | 0.86               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect     | Resource LwM2M SET                       | 1      | 0      | OK     | 0.54               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Connect to WiFi                          | 1      | 0      | OK     | 3.81               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Firmware Download                        | 1      | 0      | OK     | 90.4               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Firmware Prepare                         | 1      | 0      | OK     | 0.62               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Firmware Update                          | 1      | 0      | OK     | 31.0               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Format FAT                               | 1      | 0      | OK     | 0.35               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Initialize SD+FAT                        | 1      | 0      | OK     | 0.05               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Initialize Simple PDMC                   | 1      | 0      | OK     | 3.74               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Pelion DM Bootstrap & Reg.               | 1      | 0      | OK     | 15.85              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Pelion DM Directory                      | 1      | 0      | OK     | 6.41               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Pelion DM Re-register                    | 1      | 0      | OK     | 6.92               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update      | Post-update Identity                     | 1      | 0      | OK     | 2.46               |
+-------------------------+---------------------+-----------------------------------------------------+------------------------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 54 OK
mbedgt: completed in 892.01 sec

screamerbg commented 5 years ago

The Networking & Filesystem stress test is aiming to help identify hardware configurations where multiple functionalities might be conflicting with each other.

Here's an example for DISCO_F413ZH with SD card shield attached on the SPI Arduino pins (D10-D13), which conflicts with the ISM43362 WiFi module wired to the same pins (+ few more):

+------------------+---------------+-----------------------------------------------------+------------------------------------------+--------+--------+---------+--------------------+
| target           | platform_name | test suite                                          | test case                                | passed | failed | result  | elapsed_time (sec) |
+------------------+---------------+-----------------------------------------------------+------------------------------------------+--------+--------+---------+--------------------+
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff     1                | 1      | 0      | OK      | 11.83              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff     4                | 1      | 0      | OK      | 10.42              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff    16                | 1      | 0      | OK      | 10.06              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff    64                | 1      | 0      | OK      | 9.97               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff   256                | 1      | 0      | OK      | 9.95               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff  1024                | 1      | 0      | OK      | 9.16               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff  4096                | 1      | 0      | OK      | 6.3                |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff 16384                | 1      | 0      | OK      | 4.68               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT 1 file, buff 65536                | 1      | 0      | OK      | 4.24               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-single     | SD+FAT format                            | 1      | 0      | OK      | 0.26               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 2 files, buff 1b/256b             | 1      | 0      | OK      | 21.75              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 3 files, buff 256b/1kb/4kb        | 1      | 0      | OK      | 27.49              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT 4 files, buff 1b/256b/1kb/4kb     | 1      | 0      | OK      | 40.0               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-fs-threaded   | SD+FAT format                            | 1      | 0      | OK      | 10.9               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-single    | WiFi   256 buffer                        | 1      | 0      | OK      | 5.83               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  1k buffer                          | 1      | 0      | OK      | 3.21               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  2k buffer                          | 1      | 0      | OK      | 2.45               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  4k buffer                          | 1      | 0      | OK      | 2.61               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-single    | WiFi  8k buffer                          | 1      | 0      | OK      | 2.52               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-single    | WiFi network setup                       | 1      | 0      | OK      | 3.76               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 1 thread                            | 1      | 0      | OK      | 3.42               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 2 threads                           | 1      | 0      | OK      | 7.2                |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 3 threads                           | 1      | 0      | OK      | 11.62              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi 4 threads                           | 1      | 0      | OK      | 16.17              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-net-threaded  | WiFi network setup                       | 1      | 0      | OK      | 3.75               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT format                            | 1      | 0      | OK      | 0.26               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 1 thread, dl, file seq.      | 1      | 0      | OK      | 3.24               |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 2 threads, dl, 1kb           | 0      | 1      | FAIL    | 10.03              |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 3 threads, dl, 1b, 1kb       | 0      | 0      | SKIPPED | 0.0                |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-stress-net-fs | SD+FAT+WiFi 4 threads, dl, 1b, 256b, 1kb | 0      | 0      | SKIPPED | 0.0                |
| DISCO_F413ZH-ARM | DISCO_F413ZH  | simple-mbed-cloud-client-tests-basics-stress-net-fs | WiFi network setup                       | 1      | 0      | OK      | 5.75               |
+------------------+---------------+-----------------------------------------------------+------------------------------------------+--------+--------+---------+--------------------+
mbedgt: test case results: 1 FAIL / 2 SKIPPED / 28 OK
mbedgt: completed in 380.72 sec

screamerbg commented 5 years ago

Merging to allow Mbed Partners to use the tests to validate their platforms.

MarceloSalazar commented 5 years ago

Verified all tests pass on Disco L475 IoT

+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+--------+--------------------+
| target                  | platform_name       | test suite                                         | test case                               | passed | failed | result | elapsed_time (sec) |
+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+--------+--------------------+
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff     1            | 1      | 0      | OK     | 9.34               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff     4            | 1      | 0      | OK     | 6.64               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff    16            | 1      | 0      | OK     | 5.91               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff    64            | 1      | 0      | OK     | 5.77               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff   256            | 1      | 0      | OK     | 5.73               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff  1024            | 1      | 0      | OK     | 5.47               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff  4096            | 1      | 0      | OK     | 4.89               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS 1 file, buff 16384            | 1      | 0      | OK     | 4.72               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-single     | QSPIF+LFS format                        | 1      | 0      | OK     | 0.24               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | QSPIF+LFS 2 files, buff 4b/256b         | 1      | 0      | OK     | 12.15              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | QSPIF+LFS 3 files, buff 256b/1kb/4kb    | 1      | 0      | OK     | 16.14              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-fs-threaded   | QSPIF+LFS format                        | 1      | 0      | OK     | 0.18               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi   128 buffer                       | 1      | 0      | OK     | 10.87              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi   256 buffer                       | 1      | 0      | OK     | 6.55               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi  1024 buffer                       | 1      | 0      | OK     | 3.76               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi  4096 buffer                       | 1      | 0      | OK     | 3.06               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-single    | WiFi network setup                      | 1      | 0      | OK     | 8.36               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | WiFi 1 thread                           | 1      | 0      | OK     | 3.78               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | WiFi 2 threads                          | 1      | 0      | OK     | 8.59               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-net-threaded  | WiFi network setup                      | 1      | 0      | OK     | 3.54               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS format                        | 1      | 0      | OK     | 0.2                |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS+WiFi 1 thread, dl, file seq.  | 1      | 0      | OK     | 6.55               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS+WiFi 2 threads, dl, 1kb       | 1      | 0      | OK     | 10.03              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | QSPIF+LFS+WiFi 3 threads, dl, 256b, 1kb | 1      | 0      | OK     | 15.91              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-basic-stress-net-fs | WiFi network setup                      | 1      | 0      | OK     | 3.56               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Connect to WiFi                         | 1      | 0      | OK     | 3.51               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Format LFS                              | 1      | 0      | OK     | 0.25               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Initialize QSPIF+LFS                    | 1      | 0      | OK     | 0.07               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Initialize Simple PDMC                  | 1      | 0      | OK     | 2.79               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion DM Bootstrap & Reg.              | 1      | 0      | OK     | 14.34              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion DM Directory                     | 1      | 0      | OK     | 7.11               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion DM Re-register                   | 1      | 0      | OK     | 5.94               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Post-reset Identity                     | 1      | 0      | OK     | 2.45               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M GET                      | 1      | 0      | OK     | 0.79               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M POST                     | 1      | 0      | OK     | 0.55               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M PUT                      | 1      | 0      | OK     | 0.83               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M SET                      | 1      | 0      | OK     | 0.47               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Connect to WiFi                         | 1      | 0      | OK     | 3.57               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Firmware Download                       | 1      | 0      | OK     | 89.89              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Firmware Prepare                        | 1      | 0      | OK     | 0.63               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Firmware Update                         | 1      | 0      | OK     | 15.18              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Format LFS                              | 1      | 0      | OK     | 0.23               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize QSPIF+LFS                    | 1      | 0      | OK     | 0.03               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize Simple PDMC                  | 1      | 0      | OK     | 2.81               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion DM Bootstrap & Reg.              | 1      | 0      | OK     | 13.98              |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion DM Directory                     | 1      | 0      | OK     | 6.39               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion DM Re-register                   | 1      | 0      | OK     | 5.79               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Erase                       | 1      | 0      | OK     | 0.02               |
| DISCO_L475VG_IOT01A-ARM | DISCO_L475VG_IOT01A | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Identity                    | 1      | 0      | OK     | 2.43               |
+-------------------------+---------------------+----------------------------------------------------+-----------------------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 49 OK
mbedgt: completed in 587.52 sec

PelionIoT / simple-mbed-cloud-client

Feature: Basic tests + Pelion DM test improvements #53