Add tests for DTS operating system

[TomaszAIR]

The DTS documentation provides tests results made on couple platforms[1]. For those we use DTS to flash Dasharo firmware.

What we are missing are tests for the DTS itself. We could later post them with every new DTS release and place the results in the newsletter. The scope of these tests needs to be determine.

[1] https://docs.dasharo.com/dasharo-tools-suite/documentation/#supported-hardware

[TomaszAIR]

@sulewskiprzemyslaw maybe you could help with test suite? What and how we could test on DTS with every release.

[sulewskiprzemyslaw]

@TomaszAIR first 9 test cases for DTS might be found under this link: https://docs.dasharo.com/unified-test-documentation/dasharo-compatibility/326-dasharo-tools-suite/

Of course, we should review that list and, based on the changes in DTS, propose new test cases.

[TomaszAIR]

@sulewskiprzemyslaw how much effort do we need to test DTS Dasharo deploy on supported platforms?

[pietrushnic]

To boot DTS in QEMU:

• Create persistent storage for UEFI variables:

  cp /usr/share/OVMF/OVMF_VARS.fd /tmp/OVMF_VARS.fd

• Run QEMU

  qemu-system-x86_64 \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/OVMF_VARS.fd \
  -net nic,model=e1000 -net user \
  -nographic -m 1024

• Type exit in shell and go to Boot Manager > UEFI PXEv4
• Hit Ctrl+B while entering iPXE shell
• Get IP address using DHCP

  dhcp

• Chainload DTS

  chain http://boot.3mdeb.com/dts.ipxe

[macpijan]

To boot DTS in QEMU:

(when developing DTS)

kas-container shell meta-dts/kas.yml
runqemu slirp nographic

[macpijan]

The thing is, most of the tests in DTS that can be run in QEMU, can be made in a form of self-tests like here: https://github.com/Dasharo/meta-dts/blob/main/meta-dts-distro/recipes-tests/dts-tests/dts-tests/cukinia.conf Too bad we are not pushing this idea forward when adding new stuff apart from my initial commit one year ago.

The really important tests (like updating a particular board) must still be run on dedicated HW platforms, unless we have a lot of stubs for emulating that, which also does not guarantee the success.

[pietrushnic]

@macpijan I don't think having RF and Cukinia in parallel is a good idea. I had a discussion about that with @TomaszAIR and @artur-rs.

For some reason, RF is used for OpenBMC. The community impact of RF is way way beyond Cukinia. Cukinia is simple and small, but do we really want to manage two sets of tests and frameworks in 3mdeb and Dasharo?

[macpijan]

I don't think having RF and Cukinia in parallel is a good idea.

And here we are using nothing ¯_(ツ)_/¯

I see cukinia more as a tool almost on a level of pre-commit check. It is very quick for a developer of distro, to just build -> runqemu -> cukinia and have some very basic guarantee that my image still contains what it was supposed to contain according to the requirements. Those tools are built in Yocto (no external setup required), and you edit both (specification and code) in one place.

You would still have to use RF for any more tests that require actual interaction with the DTS, or running on hardware. And you can also reuse cukinia test by running them as one of the RF suites, not bothering of implementing keywords for these checks.

I see it as a complementary tool, not competitive.

[pietrushnic]

And here we are using nothing ¯(ツ)/¯

My hope was to get commitment to Dasharo OSFV development for DTS validation purpose, but I see there is need of some strategic alignment between EFT and EST. @artur-rs @TomaszAIR cc

I see cukinia more as a tool almost on a level of pre-commit check. It is very quick for a developer of distro, to just build -> runqemu -> cukinia and have some very basic guarantee that my image still contains what it was supposed to contain according to the requirements. Those tools are built in Yocto (no external setup required), and you edit both (specification and code) in one place.

If you put things in that way, it is an entirely different light, but it is the first time I have heard such justification. Of course, I agree with your POV.

You would still have to use RF for any more tests that require actual interaction with the DTS, or running on hardware. And you can also reuse cukinia test by running them as one of the RF suites, not bothering of implementing keywords for these checks. I see it as a complementary tool, not competitive.

That story makes a lot of sense to me and I would be glad to see such approach.

[pietrushnic]

@artur-rs @TomaszAIR I guess we get back to this topic a little bit as part of UEFI Secure Boot compliance. Maybe we can push this issue forward?