More thorough automated testing

[vchernin]

From https://github.com/wwmm/easyeffects/issues/1666 it seems clear we should be testing on older PipeWire versions. It shouldn't be too hard to just check for successful startup on different versions.

I see there is a whole test script that checks for succesful startup but it is outdated since it tests against very old screenshots.

Beyond that, I wonder how feasible it is to actually test the effects code, given GUI tests can be slow and somewhat complex. Are there examples of testing PipeWire/audio apps somewhere?'

Also, since issues like https://github.com/wwmm/easyeffects/issues/1668 were fixed by https://github.com/wwmm/easyeffects/commit/098c4f521e9c2a3a307e0f5fd3e927f744c2689c, it would be good to also try run valgrind/sanitzers with the tests (see https://github.com/wwmm/easyeffects/issues/1284).

[wwmm]

Beyond that, I wonder how feasible it is to actually test the effects code, given GUI tests can be slow and somewhat complex. Are there examples of testing PipeWire/audio apps somewhere?

I do not remember having seen examples. In the past we did some simple tests for PulseEffects. But as I did not figure out how to start PipeWire in an Arch Linux image that does not have systemd I gave up on the idea and I just compile. I like to use the circleci image based on Arch Linux so I have an automated build that has the latest version of all the packages by default. But the fact it does not use systemd is annoying when trying to start PipeWire. Pulseaudio was way easier to handle in this situation.

[vchernin]

But the fact it does not use systemd is annoying when trying to start PipeWire.

At least I was able to run pipewire from the build directory from within toolbox (podman based container). I would link to the documentation but freedesktop gitlab is down. I wonder if that actually needs systemd.

It seems podman supports using systemd better than docker. I guess an issue is circle ci may just always assume to use docker for containers, but there's probably a workaround or maybe we can use github actions instead.

[wwmm]

I wonder if that actually needs systemd.

I do not remember exactly what happened. I think that the problem was starting dbus manually in a way that PipeWire could use it. With systemd handling dbus the thing just works. But starting dbus manually in the same way I used to do for Pulseaudio did not work.

I guess an issue is circle ci may just always assume to use docker for containers, but there's probably a workaround or maybe we can use github actions instead.

It uses docker indeed. I do not have a reason to keep using circleci. We can do the same with github actions if it is possible. At this moment all that the circleci profile is doing is compiling EE in an Arch Linux docker image and running ninja install after it.

[vchernin]

At this moment all that the circleci profile is doing is compiling EE in an Arch Linux docker image and running ninja install after it.

Yeah, which is already done by the arch linux build on github actions now. When I have time I will try make the pipewire daemon work in github actions, hopefully I can get dbus to start properly.

[vchernin]

At least locally I managed to put together a script to run easyeffects (flatpak!) in podman with systemd and pipewire and dbus all more or less working. With this I can do a basic test like checking if the process shows up (and hopefully soon take a screenshot). Hopefully it is not too much extra work to get it working in github actions.

I was looking at the circle ci config and I'm not sure what the idea behind the tests is. It loads a bunch of presets/plugins, simulates devices, plays some audio but then what? How does it know if it worked properly? Is it just testing if the app crashes or not while all this is happening?

One idea I had was to play some prerecorded audio bits, then run it through some audio plugins with EE, then check if the result is "correct". But I think there would be some element of randomness in the result given EE's processsing depends on the CPU availablility etc. Do you think there is a way we could test the behaviour of the plugins somehow?

[wwmm]

Is it just testing if the app crashes or not while all this is happening?

Yes. There are many crashes that only happen once the pipeline is processing buffers. So this kind of test already helps a lot.

One idea I had was to play some prerecorded audio bits, then run it through some audio plugins with EE, then check if the result is "correct". But I think there would be some element of randomness in the result given EE's processsing depends on the CPU availablility etc. Do you think there is a way we could test the behaviour of the plugins somehow?

I think that implementing this in a way that is actually helpful can be quite challenging:

• As EasyEffects does not generate files we will have to put an audio recorder monitoring the output of the virtual sink EasyEffects will be playing to when testing the output pipeline. And for the microphone pipeline we will need to develop a different solution. Some kind of virtual source will be needed and the audio recorder will have to monitor EasyEffects virtual source.
• Depending on when the audio recorder is started there may be silence in the beginning of the recorded signal, at its end or on both. Synchronizing this may be annoying.
• There is also the question about how to better analyse the resulting file. I am not sure if comparing each sample to a reference file is the best procedure.
• Even if we find a way to handle the points above there is the fact that each plugin will need its own kind of test. It does not make much sense to test filters like the gate and the echo canceller together with the deesser or the autogain. It may be hard to analyse the result.
• Some plugins like the equalizer have very different modes of operations. The equalizer in particular would need a test that takes into account the low-pass and high-pass mode as well as the bell modes, etc. A similar thing for the multiband variants of some plugins.

It is a lot to do. And I am not sure we really need to go through all this trouble. I think that targeting crashes should be our focus. It is easier to test them and it is also easier to understand the results.

[wwmm]

On a side note I have to take some time to understand how the magic inside CI.yaml is happening :smile: . I look at it and I wonder how does it even know the name of the pkgbuild it has to use :smile:

[vchernin]

On a side note I have to take some time to understand how the magic inside CI.yaml is happening 😄 . I look at it and I wonder how does it even know the name of the pkgbuild it has to use 😄

Yes CI configs become practically their own language with the many details you have to remember…

I’m sure more comments could be added, but for the PKGBUILD it uses the one in the root directory here (that I modified slightly to make it work).

[vchernin]

It is a lot to do. And I am not sure we really need to go through all this trouble. I think that targeting crashes should be our focus. It is easier to test them and it is also easier to understand the results.

It is undoubtedly a lot to do, and I think for my initial tests I will just focus on crashes, but long term I think we should try add more involved tests. As inevitably more effects and features get added it gets more difficult to test them properly manually. At least once you’ve put in the work to figure out how to test at least some effects, the tests shouldn’t be too difficult to maintain.

[vchernin]

After spending some more time on this I think I have a clearer idea how the test running should be designed, but I still need to implement the full logic when I have time.

The main idea was to run the flatpak in a container with arbitrary pipewire daemon versions (since that is the most common issue that is annoying to test for manually). But besides that, it would let you run with the pipewire daemon and build (flatpak or non flatpak) from the host, e.g. to run the arch build with the arch version of pipewire.

The issue is that we need to mess with the host's instance of easyeffects and pipewire to run the tests. Also this wouldn't be trivially reproducible if we run the test in CI and you try to reproduce it locally (how do we know the dependencies are the same?).

Do you think it would be useful to run the tests with the arch build in a container? Are their distros besides arch it would be useful to test in a reproducible way?

[wwmm]

Do you think it would be useful to run the tests with the arch build in a container?

In the past I did this with PulseEffects in circleci. But when I tried to start PipeWire manually I was not able to make it see the dbus service. And as a result it failed.

Are their distros besides arch it would be useful to test in a reproducible way?

I think it is enough to use Arch.