COMXX CMD timeout: set --retcode true

salil0907 commented 5 years ago

Description

When trying to run test_cmdline I am getting this error 12:09:43.006 | TC MainThread: Test Case fails because of: COM55 CMD timeout: set --retcode true 12:09:43.006 | TC MainThread: Test 'test_something' FAIL, reason: COM55 CMD timeout: set --retcode true 12:09:47.505 Test case test_something failed, No retries left.

jupe commented 5 years ago

Could you use -vv flag to get more verbose traces? Probably something goes wrong when device tries to start application. You could try also application manually using some serial terminal application. Does it respond for example if you send help command?

salil0907 commented 5 years ago

after giving -vv flag and yes it responded to help command

14:05:33.459 | D1 <-- DutThread: ================ THREAD STATS =============== 14:05:33.474 | D1 <-- DutThread: ID: 0x200015cc 14:05:33.490 | D1 <-- DutThread: Name: main_thread 14:05:33.506 | D1 <-- DutThread: State: 2 14:05:33.516 | D1 <-- DutThread: Priority: 24 14:05:33.526 | D1 <-- DutThread: Stack Size: 4096 14:05:33.530 | D1 MainThread: CMD timeout: set --retcode true 14:05:33.530 | D1 <-> MainThread: CMD set --retcode false, timeout=0, time=1545640533 14:05:33.546 | D1 <-- DutThread: Stack Space: 3152 14:05:33.546 | <-> DutThread: End DUT communication 14:05:33.546 | D1 <-> MainThread: Close port 'COM55' 14:05:33.562 | TC MainThread: Close node threads 14:05:33.562 | TC MainThread: Skip Sniffing NW data 14:05:33.562 | TC MainThread: Stop external services if any 14:05:33.562 | TC MainThread: Test Case fails because of: COM55 CMD timeout: set --retcode true 14:05:33.562 | TC MainThread: retcode: 1000, preliminaryVerdict: None 14:05:33.562 | TC MainThread: Test 'test_cmdline' FAIL, reason: COM55 CMD timeout: set --retcode true 14:05:33.562 Test bench returned return code 1000 detect GIT info by folder: 'C:\Users\ssir\Downloads\mbed-os-example-blinky-master\mbed-os\TEST_APPS\testcases\example\test_cmdline.py' all git_info: {'commitid': '2fd0c5cfbd83fce62da6308f9d64c0ab64e1f0d6', 'scm': 'unknown', 'git_path': 'TEST_APPS/testcases/example/test_cmdline.py', 'scm_path': 'unknown', 'scm_link': '', 'branch': 'mbed-os-5.10', 'url': 'https://github.com/ARMmbed/mbed-os', 'scm_group': 'unknown', 'dirty': True} 14:05:34.859 | TC MainThread: {'scm': 'unknown', 'commitId': '', 'branch': 'mbed-os-5.10', 'url': 'https://github.com/ARMmbed/mbed-os', 'scm_group': 'unknown', 'commitid': '2fd0c5cfbd83fce62da6308f9d64c0ab64e1f0d6', 'git_path': 'TEST_APPS/testcases/example/test_cmdline.py', 'scm_path': 'unknown', 'dirty': True, 'scm_link': ''} 14:05:34.953 Duration: 153.136 seconds 14:05:34.953 Test case test_cmdline failed, No retries left.

after help command

C:\Users\ssir\Downloads\mbed-os-example-blinky-master\mbed-os>icetea --help usage: icetea [-h] [--list | --listsuites | --tc TC | --suite SUITE | --version] [--status STATUS] [--group GROUP] [--testtype TESTTYPE] [--subtype SUBTYPE] [--component COMPONENT] [--feature FEATURE] [--platform_filter PLATFORM_FILTER] [--jobId JOBID] [--gitUrl GITURL] [--branch BRANCH] [--commitId COMMITID] [--buildDate BUILDDATE] [--toolchain TOOLCHAIN] [--buildUrl BUILDURL] [--campaign CAMPAIGN] [--tcdir TCDIR] [--suitedir SUITEDIR] [--cfg_file CFG_FILE] [--plugin_path PLUGIN_PATH] [--allocator ALLOCATOR] [--allocator_cfg ALLOCATOR_CFG] [--env_cfg ENV_CFG] [--repeat REPEAT] [--stop_on_failure] [--clean] [--connector CONNECTOR] [--failure_return_value] [--color] [--check_version] [--ignore_invalid_params] [--parallel_flash] [--disable_log_truncate] [--cm CM] [--json] [--export SUITE_FILE_NAME] [--log LOG] [-s] [-v] [-w] [--with_logs] [--reset [RESET]] [--iface IFACE] [--bin BIN] [--tc_cfg TC_CFG] [--type {hardware,process}] [--platform_name PLATFORM_NAME] [--putty] [--skip_setup] [--skip_case] [--skip_teardown] [--valgrind] [--valgrind_tool {memcheck,callgrind,massif}] [--valgrind_extra_params VALGRIND_EXTRA_PARAMS] [--valgrind_text | --valgrind_console] [--valgrind_track_origins] [--use_sniffer] [--my_duts MY_DUTS] [--pause_ext] [--gdb GDB | --gdbs GDBS | --vgdb VGDB] [--gdbs-port GDBS_PORT] [--pre-cmds PRE_CMDS] [--post-cmds POST_CMDS] [--baudrate BAUDRATE] [--serial_timeout SERIAL_TIMEOUT] [--serial_xonxoff] [--serial_rtscts] [--serial_ch_size SERIAL_CH_SIZE] [--serial_ch_delay CH_MODE_CH_DELAY] [--nobuf NOBUF] [--kill_putty] [--forceflash | --forceflash_once] [--interface INTERFACE]

Test Framework for Command line interface

optional arguments: -h, --help show this help message and exit --list List of available testcases(nothing else) --listsuites List of available suites --tc TC execute testcase. Give test index, name, list of indices/names, or all to execute all testcases --suite SUITE Run tests from suite json file . Can be absolute path to suite file or path relative to --suitedir. --version Show version --env_cfg ENV_CFG Use user specific environment configuration file --repeat REPEAT Repeat testcases N times --stop_on_failure Stop testruns/repeation on first failed TC --clean Clean old logs --connector CONNECTOR Connector credentials for selecting and/or generating endpoint certificates. Format should be domain[:token] where token is optional. Eg. --connector this_is_some_domain:this_is_my_token --failure_return_value Sets Icetea to return a failing code to caller if one or more tests fail during the run. Default is False --color Indicates if console logs are printed plain or with colours. Default is False for plainlogs. --check_version Enables version checks for test cases. --ignore_invalid_params Disables checks for invalid parameters. --parallel_flash Enables parallel flash. --disable_log_truncate Disable long log line truncating. Over 10000characters long lines are truncated by default. --cm CM name of module that is to be used to send results to a cloud service. --json Output results of --list as json instead of a table. --export SUITE_FILE_NAME Export list into suite template file. --valgrind_text Output as Text. Default: xml format --valgrind_console Output as Text to console. Default: xml format --gdb GDB Run specific process node with gdb (debugger). e.g. --gdb 1 --gdbs GDBS Run specific process node with gdbserver (debugger). e.g. --gdbs 1 --vgdb VGDB Run specific process node with vgdb (debugger under valgrind). e.g. --vgdb 1 --forceflash Force flashing of hardware device if binary is given. Defaults to False --forceflash_once Force flashing of hardware device if binary is given, but only once. Defaults to False

Filter arguments: Arguments used for filtering tc:s

--status STATUS Run all testcases with status --group GROUP Run all testcases that have all items in <group/subgroup> or <group,group2> in their group path. --testtype TESTTYPE Run all testcases with type --subtype SUBTYPE Run all testcases with subtype <subtype --component COMPONENT Run all testcases with component --feature FEATURE Run all testcases with feature --platform_filter PLATFORM_FILTER Run all testcases that allow platform

Run information: Information of run, such as job id and git or build information. --jobId JOBID Job Unique ID --gitUrl GITURL Set application used git url for results --branch BRANCH Set used build branch for results --commitId COMMITID Set used commit ID for results --buildDate BUILDDATE Set build date --toolchain TOOLCHAIN Set toolchain for results --buildUrl BUILDURL Set build url for results --campaign CAMPAIGN Set campaign name for results Paths: Directory and file paths for various Icetea features. --tcdir TCDIR Search for testcases in directory --suitedir SUITEDIR Search for suites in directory --cfg_file CFG_FILE Load cli parameters from file. This will overwrite parameters given before --cfg_file, but results of this will be overwritten by parameters given after this one --plugin_path PLUGIN_PATH location of file called plugins_to_load, where custom plugins are imported from. Allocator: Control allocation of resources for tests. --allocator ALLOCATOR Allocator to be used for allocating resources. Default is LocalAllocator --allocator_cfg ALLOCATOR_CFG File that contains configuration for used allocator. Test case arguments: --log LOG Store logs to specific path. Filename will be /_D.log -s, --silent Silent mode, only prints results -v, --verbose increase output verbosity (print dut traces) -w Store results to a cloud service. --with_logs Store bench.log to cloud db after run. --reset [RESET] reset device before executing test cases --iface IFACE Used NW sniffer interface name --bin BIN Used specific binary for DUTs, when process is in use. NOTE: Does not affect duts which specify their own binaries. --tc_cfg TC_CFG Testcase Configuration file --type {hardware,process} Overrides DUT type. --platform_name PLATFORM_NAME Overrides used platform. Must be found in allowed_platforms in dut configuration if allowed_platforms is defined and non-empty. --putty Open putty after TC executed --skip_setup Skip TC setUp phase --skip_case Skip TC body phase --skip_teardown Skip TC tearDown phase --valgrind Analyse nodes with valgrind (linux only) --valgrind_tool {memcheck,callgrind,massif} Valgrind tool to use. --valgrind_extra_params VALGRIND_EXTRA_PARAMS Additional command line parameters to valgrind. --valgrind_track_origins Show origins of undefined values. Default: false; Used only if the Valgrind tool is memcheck --use_sniffer Use Sniffer --my_duts MY_DUTS Use only some of duts. e.g. --my_duts 1,3 --pause_ext Pause when external device command happens --gdbs-port GDBS_PORT select gdbs port --pre-cmds PRE_CMDS Send extra commands right after DUT connection --post-cmds POST_CMDS Send extra commands right before terminating dut connection. --baudrate BAUDRATE Use user defined serial baudrate (when serial device is in use) --serial_timeout SERIAL_TIMEOUT User defined serial timeout (default 0.01) --serial_xonxoff Use software flow control --serial_rtscts Use Hardware flow control --serial_ch_size SERIAL_CH_SIZE use chunck mode with size N when writing to serial port. (default N=-1: use pre-defined mode, N=0: normal, N<0: chunk-mode with size N --serial_ch_delay CH_MODE_CH_DELAY User defined delay between characters. Used only when serial_ch_size>0. (default 0.01) --nobuf NOBUF Do not use stdio buggers in node process. --kill_putty Kill old putty/kitty processes --interface INTERFACE Network interface used in tests, unless the testcase specifies which one to use. Defaults to eth0 C:\Users\ssir\Downloads\mbed-os-example-blinky-master\mbed-os>

jupe commented 5 years ago

Can you describe how you started icetea when test fails? Whole command with prameters you uses. It looks like device application is not running/starting, or perhaps it starts slowly. We need little more details what steps you actually did to be able to help more..

salil0907 commented 5 years ago

First i compiled blinky project using this command mbed compile --target DISCO_L475VG_IOT01A --toolchain GCC_ARM

than flashed the bin file onto the board then After that i tried to run test_cmdline using this command

icetea --tc test_cmdline --tcdir C:\Users\ssir\Downloads\mbed-os-example-blinky-master\mbed-os\TEST_APPS\testcases\example --platform_name DISCO_L475VG_IOT01A

jupe commented 5 years ago

Blinky application is not compatible for this test. Probable that’s the reason why it didn’t work. Could you follow these instructions to compile simple cli-application and use binary it produces: https://github.com/ARMmbed/icetea/tree/master/examples/cliapp/mbed-os5

after build success you can run icetea test like this:

icetea --tc test_cmdline --tcdir C:\Users\ssir\Downloads\mbed-os-example-blinky-master\mbed-os\TEST_APPS\testcases\example --platform_name DISCO_L475VG_IOT01A --bin <path-to-bin-file>

note:

compile cliapp using same command than before: mbed compile --target DISCO_L475VG_IOT01A --toolchain GCC_ARM
replace <path-to-bin-file> with actul filename mbed compile creates

More details about icetea usage can be found from here: https://os.mbed.com/docs/v5.10/tools/icetea-testing-applications.html

salil0907 commented 5 years ago

Which application is compatible test_cmdline testcase on icetea?

salil0907 commented 5 years ago

Hi I tried with your application i got the same error

C:\Users\ssir\Downloads\armcpp>icetea --tc test_cmdline --tcdir C:\Users\ssir\Downloads\mbed-os-example-blinky-master\mbed-os\TEST_APPS\testcases\example --platform_name DISCO_L475VG_IOT01A --bin C:\Users\ssir\Downloads\armcpp\BUILD\DISCO_L475VG_IOT01A\GCC_ARM\armcpp.bin 15:54:58.171 Using Icetea version 1.1.0 15:54:58.171 Generating suite from command line. 15:54:58.171 Importing testcases for filtering 15:54:58.184 Filtering testcases 15:54:58.184 Suite creation complete. 15:54:58.194 Test cases prepared. 15:54:58.194 Starting suite. 15:54:58.194 Starting repeat 1 of 1 15:54:58.196 Iteration 1 of 1 15:54:58.201 15:54:58.203 START TEST CASE EXECUTION: 'test_cmdline' 15:54:58.203 15:54:58.206 | TC MainThread: Start Test case 'test_cmdline' 15:54:58.207 | TC MainThread: ====setUpTestBench==== 15:54:58.233 | TC MainThread: ====setUpTestBench==== 15:54:58.233 | TC MainThread: --------------------- 15:54:58.234 | TC MainThread: Initialize DUT's connections 15:54:58.236 | RSP MainThread: Allocating 1 duts. 15:54:58.338 | LAL MainThread: Allocated device 07640221184E6161270AF738 15:54:58.339 | RSP MainThread: Allocation successful 15:54:58.348 | D1 Thread-3: Flashing dev: 07640221184E6161270AF738 15:54:58.440 | FLS Thread-3: sendBreak to device to reboot 15:54:58.693 | FLS Thread-3: reset completed 15:55:02.433 | FLS Thread-3: sendBreak to device to reboot 15:55:02.918 | FLS Thread-3: reset completed 15:55:03.624 | FLS Thread-3: 07640221184E6161270AF738 flash success 15:55:03.625 | TC Thread-3: flash ready 15:55:03.628 | TC Thread-3: Use board DISCO_L475VG_IOT01A as "D1" (id: 07640221184E6161270AF738) 15:55:03.658 | D1 MainThread: DutSerial COM55

+----------+--------------+ | COM port | Location | +----------+--------------+ | COM55 | X = 0, Y = 0 | +----------+--------------+ 15:55:03.658 | D1 <-> MainThread: Open Connection for 'D1' using 'COM55' baudrate: 115200 15:55:03.674 | D1 <-> MainThread: Use normal serial write mode 15:55:03.690 | TC MainThread: Waiting for dut 1 to initialize. 15:55:03.690 | D1 --> DutThread: set --retcode true 15:55:05.046 | D1 <-- DutThread: 15:55:53.963 | D1 MainThread: CMD timeout: set --retcode true 15:55:53.963 | TC MainThread: ====tearDownTestBench==== 15:56:45.279 | D1 MainThread: CMD timeout: set --retcode true 15:57:36.517 | D1 MainThread: CMD timeout: set --retcode true 15:57:37.793 | TC MainThread: Test Case fails because of: COM55 CMD timeout: set --retcode true 15:57:37.793 | TC MainThread: Test 'test_cmdline' FAIL, reason: COM55 CMD timeout: set --retcode true 15:57:39.401 Test case test_cmdline failed, No retries left.

jupe commented 5 years ago

what kind of device app is this: C:\Users\ssir\Downloads\armcpp\BUILD\DISCO_L475VG_IOT01A\GCC_ARM\armcpp.bin ?

Is it something similar with this: https://github.com/ARMmbed/mbed-os/blob/master/TEST_APPS/device/exampleapp/main.cpp ?

Most probably mbed application you tried does not provide cli -interface or uses custom baudrate.

salil0907 commented 5 years ago

I just updated the source file of your pointed link with my app https://github.com/ARMmbed/icetea/tree/master/examples/cliapp/mbed-os5

yes it is similar to that.

jupe commented 5 years ago

Could you try to flash cliapp you build and run mbed sterm —baudrate 115200 —echo off it opens serial port foe you and you can try to write commands to app, does it respond if you write help and press enter there for example? More details for mbed sterm is available here: https://github.com/ARMmbed/mbed-cli#serial-terminal

jupe commented 5 years ago

actually looks a bit that application starts slowly and needs special syncronization before sending commands to it. Unfortunately simplest solution is not yet available that will allow to do sync using one extra cli parameter (—sync_start). @jonikula do we have another sync approaches yet in icetea?

jonikula commented 5 years ago

Yes, the cli initialization wait feature should be available. I'll check. There is a ticket for adding the sync_start feature as well.

jonikula commented 5 years ago

Yes, you can find documentation on the cli_ready_trigger in the docs, check tc_api.md or rst. There's also an example in examples/testcase_example_usage/sample_cli_init.py.

0xc0170 commented 3 years ago

This repository is no longer maintained and it is deprecated. As result, we are closing all issues/pull request.

ARMmbed / icetea

COMXX CMD timeout: set --retcode true #67

Description