search

Dasharo / open-source-firmware-validation

OSFV infrastructure with automated tests and scripts for managing test results
Apache License 2.0
6 stars 1 forks source link

fix issue #247 #273

Closed PLangowski closed 1 week ago

macpijan commented 2 months ago

@PLangowski Can you please provide some context on why it fixes the problem? It happens only on the APU2, or the other platforms as well (especially QEMU)?

macpijan commented 1 month ago

@PLangowski I would appreciate a response here

PLangowski commented 1 month ago

@macpijan This fixes 2 problems:

The problem seems to only appear on APU2 (I haven't checked APU3 yet, because it is not available right now).

PLangowski commented 1 month ago

@macpijan I tested it on APU3 as well and the problem did not occur there.

macpijan commented 3 weeks ago

I am still not a huge fan of this workaround.

The Reenter menu ... keyword sometimes returns the previous menu's construction despite the fact that it did reenter the menu.

Can we isolate this problem further? For instance, have a test suite stressing the problematic keyword, show what the ratio of failures is, attach logs from both PASS and FAIL scenarios.

Sometimes APU reads the Secure Boot Configuration menu title as an option, which breaks the next steps. This is why I remove it from the returned construction

If anything, this should be removed much higher in a parsing keyword (perhaps here: https://github.com/Dasharo/open-source-firmware-validation/blob/main/lib/bios/menus.robot#L102), not in this particular case. What you say here, implies that there is a problem with keyword, so it must be resolved in a more generic way. Again, If there would be logs from both PASS and FAIL cases, maybe I could advise something.

We will not progress much on quality with such approach, we need a deeper understanding of the problems, rather than adding workarounds on top of each other.

PLangowski commented 1 week ago

@macpijan I have added a test that tests the Reenter menu keyword: https://github.com/Dasharo/open-source-firmware-validation/pull/329

I ran the test on APU2, and to my surprise it passed. I then ran SBO001.001 Check Secure Boot default state (firmware) several times on the latest develop and it also works. It's quite baffling that the error no longer occurs. Maybe somehting changed on develop? I'm attaching logs.

reener-menu-test.txt sb-default-state.txt