search

d-kato / esp32-driver

This repository is no longer maintained and has been moved to ARMmbed.
https://github.com/ARMmbed/esp32-driver
4 stars 14 forks source link

Mbed OS Socket test failures #4

Open VeijoPesonen opened 5 years ago

VeijoPesonen commented 5 years ago

Description

Tested the driver by executing the Mbed OS greentea network test cases.

For further details about the tests please see 

mbedgt: test case report:
| target            | platform_name | test suite              | test case                             | passed | failed | result | elapsed_time (sec) |
|-------------------|---------------|-------------------------|---------------------------------------|--------|--------|--------|--------------------|
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS                      | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_CACHE                | 1      | 0      | OK     | 0.6                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_CANCEL               | 1      | 0      | OK     | 5.32               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_EXTERNAL_EVENT_QUEUE | 1      | 0      | OK     | 3.32               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_INVALID_HOST         | 1      | 0      | OK     | 1.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_NON_ASYNC_AND_ASYNC  | 1      | 0      | OK     | 0.7                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_SIMULTANEOUS         | 1      | 0      | OK     | 1.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_SIMULTANEOUS_CACHE   | 1      | 0      | OK     | 0.99               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_SIMULTANEOUS_REPEAT  | 1      | 0      | OK     | 4.62               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | ASYNCHRONOUS_DNS_TIMEOUTS             | 1      | 0      | OK     | 2.7                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | SYNCHRONOUS_DNS                       | 1      | 0      | OK     | 0.12               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | SYNCHRONOUS_DNS_INVALID               | 1      | 0      | OK     | 1.71               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-dns     | SYNCHRONOUS_DNS_MULTIPLE              | 1      | 0      | OK     | 0.6                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS                | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS_INVALID        | 0      | 1      | FAIL   | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS_NULL           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS_PORT           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_PORT                   | 1      | 0      | OK     | 0.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_PORT_FAIL              | 0      | 1      | FAIL   | 0.09               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_UNOPENED               | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_BIND_WRONG_TYPE             | 0      | 1      | FAIL   | 0.08               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_CONNECT_INVALID             | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_ECHOTEST                    | 1      | 0      | OK     | 7.69               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_ECHOTEST_BURST              | 1      | 0      | OK     | 4.33               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_ECHOTEST_BURST_NONBLOCK     | 1      | 0      | OK     | 4.42               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_ECHOTEST_NONBLOCK           | 1      | 0      | OK     | 3.12               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_ENDPOINT_CLOSE              | 0      | 1      | FAIL   | 0.7                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_OPEN_CLOSE_REPEAT           | 1      | 0      | OK     | 0.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_OPEN_DESTRUCT               | 1      | 0      | OK     | 0.09               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_OPEN_LIMIT                  | 1      | 0      | OK     | 0.2                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_OPEN_TWICE                  | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_RECV_100K                   | 1      | 0      | OK     | 80.64              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_RECV_100K_NONBLOCK          | 1      | 0      | OK     | 82.92              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_RECV_TIMEOUT                | 1      | 0      | OK     | 1.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_SEND_REPEAT                 | 1      | 0      | OK     | 0.7                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_SEND_TIMEOUT                | 1      | 0      | OK     | 0.3                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_SETSOCKOPT_KEEPALIVE_VALID  | 0      | 1      | FAIL   | 0.6                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tcp     | TCPSOCKET_THREAD_PER_SOCKET_SAFETY    | 1      | 0      | OK     | 1.61               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_CONNECT_INVALID             | 1      | 0      | OK     | 0.8                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_ECHOTEST                    | 1      | 0      | OK     | 3.02               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_ECHOTEST_BURST              | 1      | 0      | OK     | 4.51               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_ECHOTEST_BURST_NONBLOCK     | 1      | 0      | OK     | 4.43               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_ECHOTEST_NONBLOCK           | 1      | 0      | OK     | 3.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_ENDPOINT_CLOSE              | 1      | 0      | OK     | 0.97               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_HANDSHAKE_INVALID           | 1      | 0      | OK     | 0.41               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_NO_CERT                     | 1      | 0      | OK     | 0.4                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_OPEN_DESTRUCT               | 1      | 0      | OK     | 0.01               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_OPEN_LIMIT                  | 1      | 0      | OK     | 0.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_OPEN_TWICE                  | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_RECV_TIMEOUT                | 1      | 0      | OK     | 1.31               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_SEND_CLOSED                 | 1      | 0      | OK     | 0.72               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_SEND_REPEAT                 | 1      | 0      | OK     | 1.38               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_SEND_TIMEOUT                | 1      | 0      | OK     | 0.91               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-tls     | TLSSOCKET_SEND_UNCONNECTED            | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS                | 1      | 0      | OK     | 0.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS_INVALID        | 1      | 0      | OK     | 0.13               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS_NULL           | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS_PORT           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_PORT                   | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_PORT_FAIL              | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_UNOPENED               | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_BIND_WRONG_TYPE             | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_ECHOTEST                    | 1      | 0      | OK     | 2.32               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_ECHOTEST_BURST              | 1      | 0      | OK     | 194.29             |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_ECHOTEST_BURST_NONBLOCK     | 0      | 1      | FAIL   | 3.73               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_ECHOTEST_NONBLOCK           | 1      | 0      | OK     | 6.42               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_OPEN_CLOSE_REPEAT           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_OPEN_DESTRUCT               | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_OPEN_LIMIT                  | 1      | 0      | OK     | 0.21               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_OPEN_TWICE                  | 1      | 0      | OK     | 0.0                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_RECV_TIMEOUT                | 1      | 0      | OK     | 0.91               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_SENDTO_INVALID              | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_SENDTO_REPEAT               | 1      | 0      | OK     | 0.81               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-netsocket-udp     | UDPSOCKET_SENDTO_TIMEOUT              | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-interface | NETWORKINTERFACE_CONN_DISC_REPEAT     | 1      | 0      | OK     | 5.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-interface | NETWORKINTERFACE_STATUS               | 1      | 0      | OK     | 13.98              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-interface | NETWORKINTERFACE_STATUS_GET           | 1      | 0      | OK     | 11.13              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-interface | NETWORKINTERFACE_STATUS_NONBLOCK      | 1      | 0      | OK     | 5.14               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT                          | 1      | 0      | OK     | 6.02               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-DISCONNECT-REPEAT        | 1      | 0      | OK     | 12.14              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-NOCREDENTIALS            | 0      | 2      | FAIL   | 17.99              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-PARAMS-CHANNEL           | 1      | 0      | OK     | 0.08               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-PARAMS-CHANNEL-FAIL      | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-PARAMS-NULL              | 0      | 2      | FAIL   | 30.17              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-PARAMS-VALID-SECURE      | 1      | 0      | OK     | 3.02               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-PARAMS-VALID-UNSECURE    | 1      | 0      | OK     | 3.01               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-SECURE                   | 1      | 0      | OK     | 2.02               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONNECT-SECURE-FAIL              | 1      | 0      | OK     | 17.94              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-CONSTRUCTOR                      | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-GET-RSSI                         | 1      | 0      | OK     | 4.32               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-SCAN                             | 1      | 0      | OK     | 2.21               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-SCAN-NULL                        | 1      | 0      | OK     | 2.31               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-SET-CHANNEL                      | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | tests-network-wifi      | WIFI-SET-CREDENTIAL                   | 0      | 7      | FAIL   | 0.28               |
mbedgt: test case results: 9 FAIL / 85 OK
mbedgt: completed in 876.77 sec

Version

ccbe1ed

Steps

mbed test --compile -t GCC_ARM -m GR_LYCHEE --app-config tools/test_configs/WiFiInterface.json -n tests-net*

mbedgt --test-spec=BUILD/tests/GR_LYCHEE/GCC_ARM/test_spec.json -V -n tests-netsocket-*,tests-network-wifi,tests-network-interface

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug
SeppoTakalo commented 5 years ago

Please change the title to "Mbed OS Socket test failures"

d-kato commented 5 years ago

@VeijoPesonen @SeppoTakalo, I am checking the contents, but there was a test that I was worried about. The following test "Expected return code" seems to be wrong. WIFI_SET_CREDENTIAL

Parameters Password Security Expected return code
ssid="OK" "12345678" NSAPI_SECURITY_WEP NSAPI_ERROR_OK or NSAPI_ERROR_UNSUPPORTED

The WEP only accepts passwords of 5 or 13 characters. How about the following test.

Parameters Password Security Expected return code
ssid="OK" "12345" NSAPI_SECURITY_WEP NSAPI_ERROR_OK or NSAPI_ERROR_UNSUPPORTED
SeppoTakalo commented 5 years ago

WEP is 10 or 26 hexadecimal characters. Which is equivalent of 5 or 13 bytes in binary.

WifiInterface::set_credentials() take strings, so we can assume those to be hexadecimal characters. Feeding in a binary would require length to be told also.

If this particular module accept only ascii character set, then it further limits the security of WEP by limiting the range of possible keys. I'm not sure if this is true. Please check the data sheet.

If the device expects ascii, then convert hexadecimals to ascii, and hope that it would not break the AT parser of the receiving end.

d-kato commented 5 years ago

I understood. Assume the following.

So, in the case of 5 to 26 characters, return NSAPI_ERROR_OK.

d-kato commented 5 years ago

@VeijoPesonen @SeppoTakalo, Fix Mbed OS Socket test failures https://github.com/d-kato/esp32-driver/commit/d897ebdb4423c7acb639f655a6e16c29688adeee

+-------------------+---------------+---------------------------------+---------------------------------------+--------+--------+--------+--------------------+
| target            | platform_name | test suite                      | test case                             | passed | failed | result | elapsed_time (sec) |
+-------------------+---------------+---------------------------------+---------------------------------------+--------+--------+--------+--------------------+
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS                      | 1      | 0      | OK     | 0.35               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_CACHE                | 1      | 0      | OK     | 0.64               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_CANCEL               | 1      | 0      | OK     | 5.33               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_EXTERNAL_EVENT_QUEUE | 1      | 0      | OK     | 3.51               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_INVALID_HOST         | 1      | 0      | OK     | 1.44               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_NON_ASYNC_AND_ASYNC  | 1      | 0      | OK     | 1.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_SIMULTANEOUS         | 1      | 0      | OK     | 1.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_SIMULTANEOUS_CACHE   | 1      | 0      | OK     | 0.94               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_SIMULTANEOUS_REPEAT  | 1      | 0      | OK     | 6.45               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | ASYNCHRONOUS_DNS_TIMEOUTS             | 1      | 0      | OK     | 2.83               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | SYNCHRONOUS_DNS                       | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | SYNCHRONOUS_DNS_INVALID               | 1      | 0      | OK     | 2.04               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-dns     | SYNCHRONOUS_DNS_MULTIPLE              | 1      | 0      | OK     | 0.84               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS                | 1      | 0      | OK     | 0.08               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS_INVALID        | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS_NULL           | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_ADDRESS_PORT           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_PORT                   | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_PORT_FAIL              | 1      | 0      | OK     | 0.04               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_UNOPENED               | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_BIND_WRONG_TYPE             | 1      | 0      | OK     | 0.07               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_CONNECT_INVALID             | 1      | 0      | OK     | 0.38               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_ECHOTEST                    | 1      | 0      | OK     | 8.32               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_ECHOTEST_BURST              | 1      | 0      | OK     | 9.44               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_ECHOTEST_BURST_NONBLOCK     | 1      | 0      | OK     | 9.61               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_ECHOTEST_NONBLOCK           | 1      | 0      | OK     | 8.69               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_ENDPOINT_CLOSE              | 1      | 0      | OK     | 0.69               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_OPEN_CLOSE_REPEAT           | 1      | 0      | OK     | 0.07               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_OPEN_DESTRUCT               | 1      | 0      | OK     | 0.04               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_OPEN_LIMIT                  | 1      | 0      | OK     | 0.22               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_OPEN_TWICE                  | 1      | 0      | OK     | 0.04               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_RECV_100K                   | 1      | 0      | OK     | 7.85               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_RECV_100K_NONBLOCK          | 1      | 0      | OK     | 7.3                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_RECV_TIMEOUT                | 1      | 0      | OK     | 2.44               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_SEND_REPEAT                 | 1      | 0      | OK     | 0.97               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_SEND_TIMEOUT                | 1      | 0      | OK     | 0.51               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_SETSOCKOPT_KEEPALIVE_VALID  | 1      | 0      | OK     | 0.39               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tcp     | TCPSOCKET_THREAD_PER_SOCKET_SAFETY    | 1      | 0      | OK     | 5.18               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_CONNECT_INVALID             | 1      | 0      | OK     | 1.77               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_ECHOTEST                    | 1      | 0      | OK     | 12.01              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_ECHOTEST_BURST              | 1      | 0      | OK     | 13.07              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_ECHOTEST_BURST_NONBLOCK     | 1      | 0      | OK     | 11.03              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_ECHOTEST_NONBLOCK           | 1      | 0      | OK     | 9.66               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_ENDPOINT_CLOSE              | 1      | 0      | OK     | 1.78               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_HANDSHAKE_INVALID           | 1      | 0      | OK     | 0.59               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_NO_CERT                     | 1      | 0      | OK     | 0.92               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_OPEN_DESTRUCT               | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_OPEN_LIMIT                  | 1      | 0      | OK     | 0.2                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_OPEN_TWICE                  | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_RECV_TIMEOUT                | 1      | 0      | OK     | 3.52               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_SEND_CLOSED                 | 1      | 0      | OK     | 1.74               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_SEND_REPEAT                 | 1      | 0      | OK     | 2.39               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_SEND_TIMEOUT                | 1      | 0      | OK     | 1.86               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-tls     | TLSSOCKET_SEND_UNCONNECTED            | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS                | 1      | 0      | OK     | 0.09               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS_INVALID        | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS_NULL           | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_ADDRESS_PORT           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_PORT                   | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_PORT_FAIL              | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_UNOPENED               | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_BIND_WRONG_TYPE             | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_ECHOTEST                    | 1      | 0      | OK     | 7.97               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_ECHOTEST_BURST              | 1      | 0      | OK     | 15.42              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_ECHOTEST_BURST_NONBLOCK     | 1      | 0      | OK     | 9.39               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_ECHOTEST_NONBLOCK           | 1      | 0      | OK     | 8.79               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_OPEN_CLOSE_REPEAT           | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_OPEN_DESTRUCT               | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_OPEN_LIMIT                  | 1      | 0      | OK     | 0.22               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_OPEN_TWICE                  | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_RECV_TIMEOUT                | 1      | 0      | OK     | 1.25               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_SENDTO_INVALID              | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_SENDTO_REPEAT               | 1      | 0      | OK     | 0.75               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-netsocket-udp     | UDPSOCKET_SENDTO_TIMEOUT              | 1      | 0      | OK     | 0.15               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-interface | NETWORKINTERFACE_CONN_DISC_REPEAT     | 1      | 0      | OK     | 6.14               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-interface | NETWORKINTERFACE_STATUS               | 1      | 0      | OK     | 13.38              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-interface | NETWORKINTERFACE_STATUS_GET           | 1      | 0      | OK     | 12.13              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-interface | NETWORKINTERFACE_STATUS_NONBLOCK      | 1      | 0      | OK     | 7.13               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT                          | 1      | 0      | OK     | 6.09               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-DISCONNECT-REPEAT        | 1      | 0      | OK     | 12.19              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-NOCREDENTIALS            | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-PARAMS-CHANNEL           | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-PARAMS-CHANNEL-FAIL      | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-PARAMS-NULL              | 1      | 0      | OK     | 0.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-PARAMS-VALID-SECURE      | 1      | 0      | OK     | 2.11               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-PARAMS-VALID-UNSECURE    | 1      | 0      | OK     | 2.08               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-SECURE                   | 1      | 0      | OK     | 1.05               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONNECT-SECURE-FAIL              | 1      | 0      | OK     | 15.05              |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-CONSTRUCTOR                      | 1      | 0      | OK     | 0.06               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-GET-RSSI                         | 1      | 0      | OK     | 4.27               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-SCAN                             | 1      | 0      | OK     | 1.77               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-SCAN-NULL                        | 1      | 0      | OK     | 6.02               |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-SET-CHANNEL                      | 1      | 0      | OK     | 0.1                |
| GR_LYCHEE-GCC_ARM | GR_LYCHEE     | mbed-os-tests-network-wifi      | WIFI-SET-CREDENTIAL                   | 1      | 0      | OK     | 0.05               |
+-------------------+---------------+---------------------------------+---------------------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 94 OK
mbedgt: completed in 470.74 sec

Sometimes the connection to the echo server does not work well. It may not matter, but changing line 37 of tcpsoket_echotest.cpp looks like it has been improved.

TCPSocket sock;
->
static TCPSocket sock;
SeppoTakalo commented 5 years ago

Excellent work.

@VeijoPesonen Can you verify the results in our network as well.

VeijoPesonen commented 5 years ago

Seems to be that there are still some issues left. Of course it might just be that the connection to AP is bad enough.


[1555030976.98][GLRM][INF] remote resources reset...
[1555030980.36][GLRM][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1555030980.36][CONN][INF] sending up to 10 __sync packets (specified with --sync=10)
[1555030980.36][CONN][INF] sending preamble 'ad6c6a27-c7b7-4838-8750-51e6347e9569'
[1555030980.48][GLRM][TXD] {{__sync;ad6c6a27-c7b7-4838-8750-51e6347e9569}}
[1555030980.61][CONN][RXD] mbedmbedmbedmbedmbedmbedmbedmbed
[1555030980.61][CONN][INF] found SYNC in stream: {{__sync;ad6c6a27-c7b7-4838-8750-51e6347e9569}} it is #0 sent, queued...
[1555030980.61][CONN][INF] found KV pair in stream: {{__version;1.3.0}}, queued...
[1555030980.61][CONN][INF] found KV pair in stream: {{__timeout;360}}, queued...
[1555030980.61][HTST][INF] sync KV found, uuid=ad6c6a27-c7b7-4838-8750-51e6347e9569, timestamp=1555030980.609222
[1555030980.61][HTST][INF] DUT greentea-client version: 1.3.0
[1555030980.61][HTST][INF] setting timeout to: 360 sec
[1555030980.71][CONN][RXD] >>> Running 16 test cases...
[1555030980.71][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued...
[1555030980.71][HTST][INF] host test class: '<class 'mbed_os_tools.test.host_tests.default_auto.DefaultAuto'>'
[1555030980.71][HTST][INF] host test setup() call...
[1555030980.71][HTST][INF] CALLBACKs updated
[1555030980.71][HTST][INF] host test detected: default_auto
[1555030980.81][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONSTRUCTOR}}, queued...
[1555030980.81][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-NOCREDENTIALS}}, queued...
[1555030980.81][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-SET-CREDENTIAL}}, queued...
[1555030980.91][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-SET-CHANNEL}}, queued...
[1555030980.91][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-PARAMS-NULL}}, queued...
[1555030981.01][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-SCAN-NULL}}, queued...
[1555030981.01][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-SCAN}}, queued...
[1555030981.01][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-GET-RSSI}}, queued...
[1555030981.11][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-PARAMS-VALID-UNSECURE}}, queued...
[1555030981.11][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT}}, queued...
[1555030981.21][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-DISCONNECT-REPEAT}}, queued...
[1555030981.21][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-PARAMS-VALID-SECURE}}, queued...
[1555030981.31][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-PARAMS-CHANNEL}}, queued...
[1555030981.31][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-PARAMS-CHANNEL-FAIL}}, queued...
[1555030981.41][CONN][RXD] 
[1555030981.41][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-SECURE}}, queued...
[1555030981.41][CONN][INF] found KV pair in stream: {{__testcase_name;WIFI-CONNECT-SECURE-FAIL}}, queued...
[1555030981.51][CONN][RXD] >>> Running case #1: 'WIFI-CONSTRUCTOR'...
[1555030981.51][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONSTRUCTOR}}, queued...
[1555030981.61][CONN][RXD] >>> 'WIFI-CONSTRUCTOR': 1 passed, 0 failed
[1555030981.61][CONN][RXD] 
[1555030981.61][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONSTRUCTOR;1;0}}, queued...
[1555030981.71][CONN][RXD] >>> Running case #2: 'WIFI-CONNECT-NOCREDENTIALS'...
[1555030981.71][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-NOCREDENTIALS}}, queued...
[1555030981.81][CONN][RXD] >>> 'WIFI-CONNECT-NOCREDENTIALS': 1 passed, 0 failed
[1555030981.81][CONN][RXD] 
[1555030981.81][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-NOCREDENTIALS;1;0}}, queued...
[1555030981.91][CONN][RXD] >>> Running case #3: 'WIFI-SET-CREDENTIAL'...
[1555030981.91][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-SET-CREDENTIAL}}, queued...
[1555030982.01][CONN][RXD] >>> 'WIFI-SET-CREDENTIAL': 1 passed, 0 failed
[1555030982.01][CONN][RXD] 
[1555030982.01][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-SET-CREDENTIAL;1;0}}, queued...
[1555030982.11][CONN][RXD] >>> Running case #4: 'WIFI-SET-CHANNEL'...
[1555030982.11][CONN][RXD] :35::IGNORE: set_channel() not supported
[1555030982.12][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-SET-CHANNEL}}, queued...
[1555030982.22][CONN][RXD] >>> 'WIFI-SET-CHANNEL': 1 passed, 0 failed
[1555030982.22][CONN][RXD] 
[1555030982.22][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-SET-CHANNEL;1;0}}, queued...
[1555030982.31][CONN][RXD] >>> Running case #5: 'WIFI-CONNECT-PARAMS-NULL'...
[1555030982.31][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-PARAMS-NULL}}, queued...
[1555030982.42][CONN][RXD] >>> 'WIFI-CONNECT-PARAMS-NULL': 1 passed, 0 failed
[1555030982.42][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-PARAMS-NULL;1;0}}, queued...
[1555030982.56][CONN][RXD] 
[1555030982.56][CONN][RXD] >>> Running case #6: 'WIFI-SCAN-NULL'...
[1555030982.56][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-SCAN-NULL}}, queued...
[1555030989.25][CONN][RXD] >>> 'WIFI-SCAN-NULL': 1 passed, 0 failed
[1555030989.25][CONN][RXD] 
[1555030989.25][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-SCAN-NULL;1;0}}, queued...
[1555030989.35][CONN][RXD] >>> Running case #7: 'WIFI-SCAN'...
[1555030989.35][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-SCAN}}, queued...
[1555030991.66][CONN][RXD] >>> 'WIFI-SCAN': 1 passed, 0 failed
[1555030991.66][CONN][RXD] 
[1555030991.66][CONN][RXD] >>> Running case #8: 'WIFI-GET-RSSI'...
[1555030991.66][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-SCAN;1;0}}, queued...
[1555030991.76][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-GET-RSSI}}, queued...
[1555031008.74][CONN][RXD] :36::FAIL: Expected 0 Was -3004
[1555031011.75][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031011.85][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-GET-RSSI;0;2}}, queued...
[1555031011.95][CONN][RXD] >>> 'WIFI-GET-RSSI': 0 passed, 2 failed with reason 'Test Cases Failed'
[1555031011.95][CONN][RXD] 
[1555031011.95][CONN][RXD] >>> Running case #9: 'WIFI-CONNECT-PARAMS-VALID-UNSECURE'...
[1555031012.05][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-PARAMS-VALID-UNSECURE}}, queued...
[1555031027.12][CONN][RXD] :31::FAIL: Expected 0 Was -3004
[1555031027.12][CONN][RXD] :32::FAIL: Expected 0 Was -3004
[1555031030.04][CONN][RXD] :33::FAIL: Expected 0 Was -3010
[1555031030.14][CONN][RXD] :34::FAIL: Expected 0 Was -3004
[1555031030.14][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-PARAMS-VALID-UNSECURE;0;4}}, queued...
[1555031030.24][CONN][RXD] >>> 'WIFI-CONNECT-PARAMS-VALID-UNSECURE': 0 passed, 4 failed with reason 'Test Cases Failed'
[1555031030.24][CONN][RXD] 
[1555031030.34][CONN][RXD] >>> Running case #10: 'WIFI-CONNECT'...
[1555031030.34][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT}}, queued...
[1555031045.41][CONN][RXD] :37::FAIL: Expected 0 Was -3004
[1555031045.41][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031048.42][CONN][RXD] :41::FAIL: Expected 0 Was -3010
[1555031048.42][CONN][RXD] :42::FAIL: Expected 0 Was -3004
[1555031063.70][CONN][RXD] :47::FAIL: Expected 0 Was -3004
[1555031063.79][CONN][RXD] :48::FAIL: Expected 0 Was -3004
[1555031063.79][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT;0;6}}, queued...
[1555031063.90][CONN][RXD] >>> 'WIFI-CONNECT': 0 passed, 6 failed with reason 'Test Cases Failed'
[1555031063.90][CONN][RXD] 
[1555031063.99][CONN][RXD] >>> Running case #11: 'WIFI-CONNECT-DISCONNECT-REPEAT'...
[1555031063.99][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-DISCONNECT-REPEAT}}, queued...
[1555031066.51][CONN][RXD] :38::FAIL: Expected 0 Was -3010
[1555031066.51][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031081.58][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031081.58][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031084.59][CONN][RXD] :38::FAIL: Expected 0 Was -3010
[1555031084.59][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031115.51][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031115.61][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031146.62][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031146.62][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031177.71][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031177.71][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031192.73][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031192.73][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031195.75][CONN][RXD] :38::FAIL: Expected 0 Was -3010
[1555031195.75][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031210.81][CONN][RXD] :38::FAIL: Expected 0 Was -3004
[1555031210.81][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031213.82][CONN][RXD] :38::FAIL: Expected 0 Was -3010
[1555031213.82][CONN][RXD] :40::FAIL: Expected 0 Was -3004
[1555031213.92][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-DISCONNECT-REPEAT;0;20}}, queued...
[1555031214.03][CONN][RXD] >>> 'WIFI-CONNECT-DISCONNECT-REPEAT': 0 passed, 20 failed with reason 'Test Cases Failed'
[1555031214.03][CONN][RXD] 
[1555031214.03][CONN][RXD] >>> Running case #12: 'WIFI-CONNECT-PARAMS-VALID-SECURE'...
[1555031214.12][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-PARAMS-VALID-SECURE}}, queued...
[1555031217.24][CONN][RXD] >>> 'WIFI-CONNECT-PARAMS-VALID-SECURE': 1 passed, 0 failed
[1555031217.24][CONN][RXD] 
[1555031217.24][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-PARAMS-VALID-SECURE;1;0}}, queued...
[1555031217.34][CONN][RXD] >>> Running case #13: 'WIFI-CONNECT-PARAMS-CHANNEL'...
[1555031217.34][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-PARAMS-CHANNEL}}, queued...
[1555031217.44][CONN][RXD] :33::IGNORE: set_channel() not supported
[1555031217.44][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-PARAMS-CHANNEL;1;0}}, queued...
[1555031217.54][CONN][RXD] >>> 'WIFI-CONNECT-PARAMS-CHANNEL': 1 passed, 0 failed
[1555031217.54][CONN][RXD] 
[1555031217.54][CONN][RXD] >>> Running case #14: 'WIFI-CONNECT-PARAMS-CHANNEL-FAIL'...
[1555031217.64][CONN][RXD] :33::IGNORE: set_channel() not supported
[1555031217.64][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-PARAMS-CHANNEL-FAIL}}, queued...
[1555031217.74][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-PARAMS-CHANNEL-FAIL;1;0}}, queued...
[1555031217.84][CONN][RXD] >>> 'WIFI-CONNECT-PARAMS-CHANNEL-FAIL': 1 passed, 0 failed
[1555031217.84][CONN][RXD] 
[1555031217.84][CONN][RXD] >>> Running case #15: 'WIFI-CONNECT-SECURE'...
[1555031217.94][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-SECURE}}, queued...
[1555031219.95][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-SECURE;1;0}}, queued...
[1555031220.05][CONN][RXD] >>> 'WIFI-CONNECT-SECURE': 1 passed, 0 failed
[1555031220.05][CONN][RXD] 
[1555031220.05][CONN][RXD] >>> Running case #16: 'WIFI-CONNECT-SECURE-FAIL'...
[1555031220.15][CONN][INF] found KV pair in stream: {{__testcase_start;WIFI-CONNECT-SECURE-FAIL}}, queued...
[1555031235.21][CONN][RXD] >>> 'WIFI-CONNECT-SECURE-FAIL': 1 passed, 0 failed
[1555031235.21][CONN][RXD] 
[1555031235.21][CONN][INF] found KV pair in stream: {{__testcase_finish;WIFI-CONNECT-SECURE-FAIL;1;0}}, queued...
[1555031235.31][CONN][RXD] >>> Test cases: 12 passed, 4 failed with reason 'Test Cases Failed'
[1555031235.31][CONN][RXD] >>> TESTS FAILED!
[1555031235.31][CONN][INF] found KV pair in stream: {{__testcase_summary;12;4}}, queued...
[1555031235.41][CONN][INF] found KV pair in stream: {{max_heap_usage;1748}}, queued...
[1555031235.41][CONN][INF] found KV pair in stream: {{reserved_heap;1851960}}, queued...
[1555031235.41][CONN][INF] found KV pair in stream: {{__thread_info;"0x0200230d8",2716,4096}}, queued...
[1555031235.41][HTST][ERR] orphan event in main phase: {{max_heap_usage;1748}}, timestamp=1555031235.412957
[1555031235.41][HTST][ERR] orphan event in main phase: {{reserved_heap;1851960}}, timestamp=1555031235.412963
[1555031235.41][HTST][ERR] orphan event in main phase: {{__thread_info;"0x0200230d8",2716,4096}}, timestamp=1555031235.412965
[1555031235.51][CONN][INF] found KV pair in stream: {{__thread_info;"0x020023228",168,512}}, queued...
[1555031235.51][CONN][INF] found KV pair in stream: {{__thread_info;"0x0200231e4",104,768}}, queued...
[1555031235.51][HTST][ERR] orphan event in main phase: {{__thread_info;"0x020023228",168,512}}, timestamp=1555031235.513171
[1555031235.51][HTST][ERR] orphan event in main phase: {{__thread_info;"0x0200231e4",104,768}}, timestamp=1555031235.513177
[1555031235.61][CONN][RXD] {{__cpu_info        up time;253072032}}
[1555031235.61][CONN][RXD] {{__cpu_info     sleep time;247305536}}
[1555031235.61][CONN][RXD] {{__cpu_info deepsleep time;0}}
[1555031235.71][CONN][RXD] {{__cpu_info  %  sleep/deep;97;0}}
[1555031235.71][CONN][INF] found KV pair in stream: {{end;failure}}, queued...
[1555031235.71][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1555031235.71][HTST][INF] __exit(0)
[1555031235.72][HTST][INF] __notify_complete(False)
[1555031235.72][HTST][INF] __exit_event_queue received
[1555031235.72][HTST][INF] test suite run finished after 255.11 sec...
michalpasztamobica commented 5 years ago

Small update: I cannot see the WIFI failures on my desk and I see that now they fail irregularly in the nightly runs, therefore I assume this might be a connectivity issue, rather than a SW bug.

New observation: Both the nightly tests and my local set up show a different mbed socket tests failure at the moment in tests-netsocket-udp:

[1560843500.22][CONN][RXD] >>> Running case #16: 'UDPSOCKET_ECHOTEST_NONBLOCK'...
[1560843500.23][CONN][INF] found KV pair in stream: {{__testcase_start;UDPSOCKET_ECHOTEST_NONBLOCK}}, queued...
[1560843532.17][CONN][RXD] Packets sent: 25, packets received 15, loss ratio 0.40
mbedgt: :205::FAIL: Values Not Within Delta
[1560843532.17][CONN][RXD] :205::FAIL: Values Not Within Delta
...
[1560843541.10][CONN][INF] found KV pair in stream: {{__testcase_finish;UDPSOCKET_SENDTO_REPEAT;1;0}}, queued...
[1560843541.11][CONN][RXD] >>> Running case #19: 'UDPSOCKET_ECHOTEST'...
[1560843541.11][CONN][INF] found KV pair in stream: {{__testcase_start;UDPSOCKET_ECHOTEST}}, queued...
[1560843542.17][CONN][RXD] [Round#15 - Receiver] error, returned 512
[1560843542.21][CONN][RXD] [Round#15 - Receiver] error, returned 88
[1560843542.29][CONN][RXD] [Round#15 - Receiver] error, returned 512
[1560843542.33][CONN][RXD] [Round#16 - Receiver] error, returned 88
[1560843542.40][CONN][RXD] [Round#16 - Receiver] error, returned 512
[1560843542.46][CONN][RXD] [Round#16 - Receiver] error, returned 88
[1560843542.53][CONN][RXD] [Round#17 - Receiver] error, returned 512
[1560843542.61][CONN][RXD] [Round#17 - Receiver] error, returned 188
[1560843542.69][CONN][RXD] [Round#17 - Receiver] error, returned 512
[1560843542.77][CONN][RXD] [Round#18 - Receiver] error, returned 188
[1560843542.84][CONN][RXD] [Round#18 - Receiver] error, returned 512
[1560843542.90][CONN][RXD] [Round#18 - Receiver] error, returned 188
[1560843542.97][CONN][RXD] [Round#19 - Receiver] error, returned 512
[1560843543.03][CONN][RXD] [Round#19 - Receiver] error, returned 288
[1560843543.10][CONN][RXD] [Round#19 - Receiver] error, returned 512
[1560843543.20][CONN][RXD] [Round#20 - Receiver] error, returned 288
[1560843543.31][CONN][RXD] [Round#20 - Receiver] error, returned 512
[1560843543.38][CONN][RXD] [Round#20 - Receiver] error, returned 288
[1560843543.51][CONN][RXD] [Round#21 - Receiver] error, returned 512
[1560843543.61][CONN][RXD] [Round#21 - Receiver] error, returned 388
[1560843543.72][CONN][RXD] [Round#21 - Receiver] error, returned 512
[1560843543.72][CONN][RXD] Packets sent: 36, packets received 15, loss ratio 0.58

I found that this commit is causing the failures: 74b547507dcee6ff6684029d8b4fa1c17321a76d

michalpasztamobica commented 5 years ago

Perhaps I made a mistake when bisecting. This change looks way more suspicious: 9817dc00529a70d4a2163347272e5fef8517cc0b . @d-kato , would you have a look at this, please?

d-kato commented 5 years ago

@michalpasztamobica , thank you. I have confirmed the problem.
I think that the cause is that by dividing the transmission, the received data is also divided.
Fixed: https://github.com/d-kato/esp32-driver/commit/4488aaeee495b1f6a240dfa9aff0904fda7fd0d8

GR_LYCHEE has no problem because it uses RTS and CTS terminals, but in the case of a board that does not use CTS, data overflows. It has been fixed as follows:

  1. When using CTS, 2048 bytes of data are sent at one time.
  2. When not using CTS, it is divided into 512 bytes and sent.

I think that it is better than not being able to send even if it is divided.