`Execute Command In Terminal` returns wrong output on Windows

[m-iwanicki]

Device

Protectli VP2420

RTE version

0.7.5 (dunfell)

OSFV version

develop

Affected component(s) or functionality

No response

Brief summary

multiple Execute Command In Terminal on Windows doesn't return only cmd output

How reproducible

Always but with different results

How to reproduce

Command used to run test:

robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.221 -v device_ip:192.168.10.161 -v config:protectli-vp2420 test.robot

test.robot:

*** Settings ***
Library             Telnet
Library             SSHLibrary
Library             RequestsLibrary
Resource            variables.robot
Resource            keywords.robot
Resource            keys.robot
Resource            sonoff-rest-api/sonoff-api.robot
Resource            rtectrl-rest-api/rtectrl.robot
Resource            pikvm-rest-api/pikvm_comm.robot

Suite Setup         Run Keyword
...                     Prepare Test Suite
Suite Teardown      Run Keyword
...                     Log Out And Close Connection

*** Test Cases ***
TEST
    Power On
    Login To Windows
    ${out}=    Execute Command In Terminal    Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
    ${out}=    Execute Command In Terminal    Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
    Log To Console    ${out}

Expected behavior

${out} should contain only output of command, e.g.

Status     Class           FriendlyName                                                                     InstanceId
------     -----           ------------                                                                     ----------
OK         USB             USB Mass Storage Device                                                          USB\VID_0781...
OK         USB             USB Root Hub (USB 3.0)                                                           USB\ROOT_HUB...
OK         USB             Qualcomm USB Composite Device 90B3                                               USB\VID_05C6...
OK         Net             Remote NDIS based Internet Sharing Device                                        USB\VID_05C6...
OK         DiskDrive        USB  SanDisk 3.2Gen1 USB Device                                                 USBSTOR\DISK...
OK         Bluetooth       Intel(R) Wireless Bluetooth(R)                                                   USB\VID_8087...

Actual behavior

${out} contains part or whole of command that was called

>> Ge
>> > Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
>>
Status     Class           FriendlyName                                                                     InstanceId
------     -----           ------------                                                                     ----------
OK         USB             USB Mass Storage Device                                                          USB\VID_0781...

What is returned is inconsistent, e.g.:

>> Ge
>> > Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match
>> > Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
>>
------     -----           ------------                                                                     ----------
OK         USB             USB Mass Storage Device                                                          USB\VID_0781...

Link to screenshots or logs

logs: log.tar.gz

Additional context

No response

Solutions you've tried

No response

[macpijan]

@BeataZdunczyk Inconsistent output from keyword leads to inconsistent tests results. We need to get to the bottom of this problem, or at least workaround this in keyword, to drop the extra content. Need reliability suite for this keyword as well to ensure this is properly improved.

[JanPrusinowski]

Fixed in: https://github.com/Dasharo/open-source-firmware-validation/pull/402 log.zip

[m-iwanicki]

@JanPrusinowski There is nothing in your PR that fixes it. Also in your logs second Execute command returns:

Return: ">> Ge\r\n>> Get-\r\n>> > Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '\r\n>> > Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }\r\n>> \r\n\r\nStatus     Class           FriendlyName                                                                     InstanceId     \r\n------     -----           ------------                                                                     ----------     \r\nOK         USB             USB Mass Storage Device                                                          USB\\VID_0781...\r\nOK         USB             USB Root Hub (USB 3.0)                                                           USB\\ROOT_HUB...\r\nOK         USB             Qualcomm USB Composite Device 90B3                                               USB\\VID_05C6...\r\nOK         Net             Remote NDIS based Internet Sharing Device                                        USB\\VID_05C6...\r\nOK         DiskDrive        USB  SanDisk 3.2Gen1 USB Device                                                 USBSTOR\\DISK...\r\nOK         Bluetooth       Intel(R) Wireless Bluetooth(R)                                                   USB\\VID_8087..."

[JanPrusinowski]

I was trying to create a keyword which would clean the output from the command and any trash that might have stayed in the console from running the previous command. https://github.com/Dasharo/open-source-firmware-validation/pull/402/commits/7889341d3860daa8e2736ab23b3445c346026bdb Unfortunatelly there is an issue when comparing two identical strings:

OUT1 START

.
STRING CONTAINS:
Line:  G> Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
Command: Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
ADD: False

STRING CONTAINS:
Line: 
Command: Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
ADD: False

STRING CONTAINS:
Line: Status     Class           FriendlyName                                                                     InstanceId     
Command: Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }
ADD: False

Command is a substring of Line - but robot framework returns check condition as false.

To run:

robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.221 -v device_ip:192.168.10.161 -v config:protectli-vp2420 -t 'TEST*' self-tests/os-boot.robot

[macpijan]

@JanPrusinowski

This fixes the issue for me: https://github.com/Dasharo/open-source-firmware-validation/pull/409/files

I focused on trying something else and fixing the original issue, more than trying to get to the bottom of weird behavior you face, as it seemed overly complex.

Please confirm.

[JanPrusinowski]

@macpijan Your fix works. logs.zip

@m-iwanicki PR with test: https://github.com/Dasharo/open-source-firmware-validation/pull/402