search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.16k stars 14.18k forks source link

Add installed tests #34987

Open jtojnar opened 6 years ago

jtojnar commented 6 years ago

We want to test that packages work on NixOS but writing tests manually is a chore so not many applications are covered. Fortunately, many packages (especially from GNOME or Freedesktop) ship so called installed tests that already do what we want.

I already implemented them for gjs and fwupd and as you can see it is actually quite simple. We can even factor it out into something like:

import ./make-test.nix ({ pkgs, ... }:
  let
    makeInstalledTest = { tested, withX ? false }: {
      name = tested.name;

      meta = {
        maintainers = tested.meta.maintainers;
      };

      machine = { pkgs, ... }: {
        imports = stdenv.lib.optional withX ./common/x11.nix;
        environment.systemPackages = with pkgs; [ gnome-desktop-testing ];
        environment.variables.XDG_DATA_DIRS = [ "${tested.installedTests}/share" ];
      };

      testScript = ''
        ${stdenv.lib.optional withX "$machine->waitForX;"}
        $machine->succeed("gnome-desktop-testing-runner");
      '';
    };
  in makeInstalledTest { tested = pkgs.gjs; withX = true; }
)

Here is incomplete list of software that provides installed tests:

Finally, we should connect the tests to their respective packages, for example using https://github.com/NixOS/nixpkgs/issues/27604

jtojnar commented 5 years ago

Ibus apparently has something based on gdtr https://github.com/fujiwarat/ibus-anthy/commit/5a9e4858f1a2b6cad02ef1d272fc5bed959ec0bf, we should probably use that to prevent things like https://github.com/NixOS/nixpkgs/issues/68837 from reoccurring.

stale[bot] commented 4 years ago

Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on irc.freenode.net.
jtojnar commented 4 years ago

This is a long term tracking goal.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

willbush commented 2 years ago

Hello! I'd like to start contributing to nixpkgs. I was thinking to try to improve testing to prevent regressions. Is this a good issue to start with?

jtojnar commented 2 years ago

@willbush Hi. It will depend on your experience and vary from package to package. For libraries that are mostly self-contained, it may be as simple as adding the tested package and making sure the installed test files are installed by the package (usually there will be an configure option, though sometimes it is broken because upstreams do not regularly run installed tests). Sometimes there are paths that need to be patched or extra dependencies to be added, just like with any other Nix packaging. With extra dependencies, it might be nice to move the installed tests into a separate output.

Here are recent commits introducing installed tests to geocode-glib as another example: https://github.com/NixOS/nixpkgs/commit/fb2877c36f3e2a6de74dce7310160d9673a4a48f https://github.com/NixOS/nixpkgs/commit/6a7da5b520eba152f200cd761e4d90e6aa404af5

json-glib, pango, libsoup, gtksourceview, evolution-data-server, evolution, tracker, gvfs, tracker-miners are the more consequential packages where tests might be useful (ordered roughly by the estimated complexity of getting the tests running, easiest first).

Feel free to ping me on Matrix or Discourse if you have more specific questions.

willbush commented 2 years ago

@jtojnar Took a while to get over the learning curve, but I think json-glib PR is inline with how others were done.

I noticed that graphene and json-glib tests execute very fast. I was wondering why a VM has to be spun up for them? Is it because we don't know how they modify their environment?

willbush commented 2 years ago

@jtojnar I noticed that gtksourceview5 runs meson test in the checkPase. Would installed tests still be desired?

jtojnar commented 2 years ago

@willbush Sorry, this got lost in the noise.

I noticed that graphene and json-glib tests execute very fast. I was wondering why a VM has to be spun up for them? Is it because we don't know how they modify their environment?

I think for installed tests it is desirable to make them closer to a real runtime environment.

I noticed that gtksourceview5 runs meson test in the checkPase. Would installed tests still be desired?

Looks like the installed tests of gtksourceview5 run the same set of tests as the test suite so installing them is probably not necessary.

I would say that the installed tests mainly make sense for tests that require realistic runtime environment or ones that take long time. If the test suite is mostly self-contained, running it in the build is sufficient.