search

bus1 / dbus-broker

Linux D-Bus Message Broker
https://github.com/bus1/dbus-broker/wiki
Apache License 2.0
661 stars 78 forks source link

test: integration tests #348

Closed mrc0mmand closed 4 months ago

mrc0mmand commented 4 months ago

Let's follow-up on 0a0b0fa and introduce a second Packit job that runs integration tests on the just built RPMs in Testing Farm [0].

Since there are no pre-existing integration tests, add a simple one that runs dfuzzer on the org.freedesktop.systemd1 D-Bus interface to exercise dbus-broker a bit.

[0] https://packit.dev/docs/configuration/upstream/tests

This is the continuation of #347 which adds an integration test "infrastructure" with a single simple test, and some notes on how to add new tests. I followed the original intention from #279 and used tmt here, but separated the tmt tree into a subdirectory, so the fmf metadata don't pollute the project root (just a personal preference, feel free to rearrange).

I'm opening this as a draft, since there's one issue with this approach - tmt, in its entirety, works properly only on RHEL, CentOS, and Fedora, since it uses dnf/yum to install packages. AFAIU this doesn't affect local runs (see https://tmt.readthedocs.io/en/stable/questions.html#using-tmt-outside-of-fedora-centos-and-rhel-distribution), so it should still be possible to run the tests to some degree, but I personally have no experience with tmt outside of the RH ecosystem. This may or may not be a an issue; on one hand there is no pre-existing integration test suite, so nothing will suddenly break, and as tmt is fully supported by the Fedora CI infra the integration with is then very simple. However, other distributions might run into some obstacles when/if they try to run it on their side. So treat this PR as a PoC that shows how it could look like, but feel free to discard if the limitations are not acceptable.

teg commented 4 months ago

Thank you for this!

I appreciate the consideration to other distros, but I think focusing on Fedora/CentOS/RHEL is fine, as integration tests are by their nature tied to the target distros. We would of course welcome contributions of integration tests for other distros if others have an interest in that.

The structure looks good to me. I think fuzzing systemd1 makes sense as we do parse all these messages and it will get good coverage for our parser.

A natural extension would be to fuzz also the dbus control interface as that's what dbus-broker itself implements, but that could be done in a follow up.

Once you are happy with this PR and mark it ready for review I'd be happy to merge it. @dvdhrm ?

mrc0mmand commented 4 months ago

Thank you for this!

I appreciate the consideration to other distros, but I think focusing on Fedora/CentOS/RHEL is fine, as integration tests are by their nature tied to the target distros. We would of course welcome contributions of integration tests for other distros if others have an interest in that.

The structure looks good to me. I think fuzzing systemd1 makes sense as we do parse all these messages and it will get good coverage for our parser.

A natural extension would be to fuzz also the dbus control interface as that's what dbus-broker itself implements, but that could be done in a follow up.

Once you are happy with this PR and mark it ready for review I'd be happy to merge it. @dvdhrm ?

I beefed up the commit message a bit, so it should be ready to go. As discussed internally, I'll look into running the reference tests next once I have a bit of spare time.

teg commented 4 months ago

The integration tests appear to be stuck?

mrc0mmand commented 4 months ago

The integration tests appear to be stuck?

There was an unexpected outage caused by a wild workaround (https://status.testing-farm.io/issues/2024-03-12-long-queue/). The issue should be resolved, but there's still a long queue Testing Farm has to chew through, so it might take a bit (but the runners seem to have already picked our jobs as I'm writing this).