Nitrokey / nitrokey-3-tests

Automated tests for the Nitrokey 3
3 stars 0 forks source link

Add helper classes for upgrade tests #11

Closed robin-nitrokey closed 1 year ago

robin-nitrokey commented 1 year ago

This PR adds helper classes that make it easier two write upgrade tests. Instead of writing the same test twice, once for a single device with the current firmware and once with a preparation step with the old firmware and a verification step with the current firmware:

def test_fido2(device) -> None:
    fido2 = Fido2(device.hidraw)
    credential = fido2.register(b"user_id", "A. User")
    fido2.authenticate([credential])

@upgrade
@pytest.mark.virtual
def test_upgrade_fido2(serial: str, ifs: str) -> None:
    with spawn_device(serial=serial, ifs=ifs, suffix="old") as device:
        fido2 = Fido2(device.hidraw)
        credential = fido2.register(b"user_id", "A. User")
    with spawn_device(serial=serial, ifs=ifs, provision=False) as device:
        fido2 = Fido2(device.hidraw)
        fido2.authenticate([credential])

We can now write it only once:

class TestFido2(ExecUpgradeTest):
    @contextmanager
    def context(self, device):
        yield Fido2(device.hidraw)

    def prepare(self, fido2):
        return fido2.register(b"user_id", "A. User")

    def run(self, fido2, credential):
        fido2.authenticate([credential])

This will automatically generate two test functions like these:

    def test_exec(self, device: Device) -> None:
        with self.context(device) as context:
            state = self.prepare(context)
            self.run(context, state)

    @upgrade
    @pytest.mark.virtual
    def test_exec_upgrade(self, serial: str, ifs: str) -> None:
        with spawn_device(serial=serial, ifs=ifs, suffix="old") as device:
            with self.context(device) as context:
                state = self.prepare(context)
        with spawn_device(serial=serial, ifs=ifs, provision=False) as device:
            with self.context(device) as context:
                self.run(context, state)

The big advantage of this approach is that we can easily test upgrade stability and data retention for all features. The downside is that we have to change the scope of the device fixture from module to function because the upgrade tests are no longer in a separate module. That means we have to spawn a new virtual device for every test function, adding an overhead of up to five seconds per test.

Which way do you prefer? Or do you have any ideas how to avoid the overhead?

robin-nitrokey commented 1 year ago

A good trade-off could be to only auto-generate the simple case and manually invoke the upgrade case in a separate module.

robin-nitrokey commented 1 year ago

And with a parametrized tests, we can even automate the upgrade case.