Rework `usb-hid-and-msc-support.robot`

Psotas commented 4 months ago

The problem you're addressing (if any)

[ ] USB001.001 USB devices detected in FW only works with PiKVM
[ ] https://github.com/Dasharo/open-source-firmware-validation/issues/87
[ ] https://github.com/Dasharo/open-source-firmware-validation/issues/206
[ ] https://github.com/Dasharo/open-source-firmware-validation/issues/279

Describe the solution you'd like

No response

Where is the value to a user, and who might that user be?

No response

Describe alternatives you've considered

No response

Additional context

No response

m-iwanicki commented 1 week ago

USB001.001 should work when this PR will be merged: https://github.com/Dasharo/open-source-firmware-validation/pull/351

Currently:

 robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.233 -v config:protectli-vp2410  dasharo-compatibility/usb-hid-and-msc-support.robot
==============================================================================
Usb-Hid-And-Msc-Support
==============================================================================
USB001.001 USB devices detected in FW :: Check whether USB devices... | FAIL |
Variable '${PIKVM_IP}' not found.
------------------------------------------------------------------------------
USB001.002 USB devices detected by OS (Ubuntu) :: Check whether th... ......
Checking if usbutils is installed...

Package usbutils is installed
USB001.002 USB devices detected by OS (Ubuntu) :: Check whether th... | FAIL |
'      bInterfaceClass         8 Mass Storage
      bInterfaceClass         9 Hub
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass       224 Wireless
      bInterfaceClass         9 Hub' does not contain 'Human Interface Device'
------------------------------------------------------------------------------
USB001.003 USB devices detected by OS (Windows) :: Check whether t... | SKIP |
USB001.003 not supported
------------------------------------------------------------------------------
USB002.001 USB keyboard detected in FW :: Check whether the extern... | PASS |
------------------------------------------------------------------------------
USB002.002 USB keyboard in OS (Ubuntu) :: Check whether the extern... | PASS |
------------------------------------------------------------------------------
USB002.003 USB keyboard in OS (Windows) :: Check whether the exter... | SKIP |
USB002.003 not supported
------------------------------------------------------------------------------
USB003.001 Upload 1GB file on USB storage (Ubuntu) :: Check whethe... | SKIP |
USB003.001 not supported
------------------------------------------------------------------------------
USB003.002 Upload 1GB file on USB storage (Windows) :: Check wheth... | SKIP |
USB003.002 not supported
------------------------------------------------------------------------------
Usb-Hid-And-Msc-Support                                               | FAIL |
8 tests, 2 passed, 2 failed, 4 skipped
==============================================================================

With my PR:

 robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.233 -v config:protectli-vp2410  dasharo-compatibility/usb-hid-and-msc-support.robot
==============================================================================
Usb-Hid-And-Msc-Support
==============================================================================
USB001.001 USB devices detected in FW :: Check whether USB devices... | PASS |
------------------------------------------------------------------------------
USB001.002 USB devices detected by OS (Ubuntu) :: Check whether th... ......
Checking if usbutils is installed...

Package usbutils is installed
USB001.002 USB devices detected by OS (Ubuntu) :: Check whether th... | PASS |
------------------------------------------------------------------------------
USB001.003 USB devices detected by OS (Windows) :: Check whether t... | SKIP |
USB001.003 not supported
------------------------------------------------------------------------------
USB002.001 USB keyboard detected in FW :: Check whether the extern... | SKIP |
USB002.001 not supported
------------------------------------------------------------------------------
USB002.002 USB keyboard in OS (Ubuntu) :: Check whether the extern... | SKIP |
USB002.002 not supported
------------------------------------------------------------------------------
USB002.003 USB keyboard in OS (Windows) :: Check whether the exter... | SKIP |
USB002.003 not supported
------------------------------------------------------------------------------
USB003.001 Upload 1GB file on USB storage (Ubuntu) :: Check whethe... | SKIP |
USB003.001 not supported
------------------------------------------------------------------------------
USB003.002 Upload 1GB file on USB storage (Windows) :: Check wheth... | SKIP |
USB003.002 not supported
------------------------------------------------------------------------------
Usb-Hid-And-Msc-Support                                               | PASS |
8 tests, 2 passed, 0 failed, 6 skipped
==============================================================================

2 failing tests now succeed and 2 succeeding tests are now skipped as on tested machine there was no keyboard.

m-iwanicki commented 1 week ago

Both should be now fixed in https://github.com/Dasharo/open-source-firmware-validation/pull/351:

USB001.001 USB devices detected in FW only works with PiKVM
https://github.com/Dasharo/open-source-firmware-validation/issues/87

From what I've tested

https://github.com/Dasharo/open-source-firmware-validation/issues/279

Should also be fixed (after merge)

m-iwanicki commented 1 week ago

usb-hid-and-msc-support.robot should still be reworked somehow, because currently I think for tests to complete successfully USB drive should have preferably one FAT32 partition that's a little over 2 GB in size (maybe a little over 1 GB would work but I'm not sure). This partition should also be "bootable" so USB is detected in Boot Menu In my tests I prepared USB with gdisk, with partition set as EFI (type EF00) and then formatted with mkfs.fat -F 32 /dev/sdc1

ubuntu@3mdeb:~$ lsblk /dev/sdc -o +fstype
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS FSTYPE
sdc      8:32   1 28,7G  0 disk
└─sdc1   8:33   1   10G  0 part             vfat

And then copied EFI directory from random /boot directory (left only Boot directory removed rest)

.
├── EFI
│   └── Boot
│       ├── bootx64.efi
│       ├── fbx64.efi
│       └── mmx64.efi
└── System Volume Information
    ├── IndexerVolumeGuid
    └── WPSettings.dat

3 directories, 5 files

I also had to set disk letter first time manually in Windows.

Dasharo / open-source-firmware-validation