ZTS: ZFS Test Suite

[Atemu]

Project description describe the project a little

ZFS has an automatic regression testing suite, it'd be great if we could run that in our ZFS NixOS tests.

It'd catch some regressions we've had when updating linux_latest before upstream is fully compatible for example which I've been hit by before.

Metadata

• homepage URL: https://github.com/openzfs/zfs/tree/master/tests
• source URL: https://github.com/openzfs/zfs/tree/master/tests
• license: CDDL
• platforms: linux

[Mindavi]

This sounds like a good idea and (in theory, reading the instructions) like it should be very doable. I might pick this up to learn more about the NixOS test driver. I'll have to see if I get around to it, though.

[Mindavi]

A first attempt at getting this to work. It appears that the ZTS test assume a FHS at first glance, that might either require some patching or running in a FHS env (don't know if we can do that though?).

Current output is:

zfs-tests.sh: /sbin/losetup utility must be installed

After that some other tools might need replacing and then we'll have to see what breaks. It doesn't seem like this is as easy as I hoped, but maybe it's okay.

# nixpkgs/nixos/tests/zfs-tests.nix
import ./make-test-python.nix ({ pkgs, ... } : {
  name = "zfs-tests";
  meta = with pkgs.stdenv.lib.maintainers; {
    maintainers = [ mindavi ];
  };

  machine =
    { lib, pkgs, config, ... }:
    with lib;
    { users.users.alice = { isNormalUser = true; group = "wheel"; };
      imports = [ ./common/user-account.nix ];
      nixpkgs.overlays = [
        (self: super: {
          zfs = super.zfs.overrideAttrs (oldAttrs: {
            postPatch = (oldAttrs.postPatch or "") + ''
              substituteInPlace scripts/zfs-tests.sh \
                --replace \$STF_PATH/ksh ${pkgs.ksh}/bin/ksh
              # Patching probably required for more binaries
              # Maybe FHS environment would be better?
              echo Patched ksh to ${pkgs.ksh}/bin/ksh
              grep "This test suite requires ksh" scripts/zfs-tests.sh
            '';
          });
        })
      ];
      virtualisation.emptyDiskImages = [ 4096 ];
      security.sudo.wheelNeedsPassword = false;
      boot.kernelModules = [ "zfs" ];
      boot.supportedFilesystems = [ "zfs" ];
      environment.systemPackages = [
        pkgs.gcc
        pkgs.zfs
        pkgs.ksh
        #pkgs.gzip
        #pkgs.gunzip
      ];
      networking.hostId = "abcdef01";
      # virtualisation.writableStore = true; # may be needed for the zfs tests
    };

  testScript = ''
      start_all()
      machine.succeed("modprobe zfs", "zfs version", "id")
      machine.succeed(
          "su - alice ${pkgs.zfs}/share/zfs/zfs-tests.sh"
      )
    '';
})

Note that this requires that the line in the zfs package pkgs/os-specific/linux/zfs/default.nix that removes the tests is removed.

[Mindavi]

I've got it (mostly?) running now, see https://github.com/Mindavi/nixpkgs/tree/feature/zfs-tests.

However, the test suite takes very long to run (I've not yet ran it to completion because it takes so long). Would you have any pointers on what tests are more critical than others? I don't think we can let hydra run tests that take more than an hour (and I don't want to take up all resources just for the zfs support).

The test suite supports selecting parts by using run files or by using tags (such as -T zdb). I think it's best to select the most critical ones and leave the others off by default, otherwise I think it'll be too much of a burden on the infrastructure.

I'll let the test suite run for today. I'll post a log here and if I find any issues I might notify upstream or the NixOS ZFS maintainers.

Thanks for the suggestion, this is interesting to look at :).

[rbrewer123]

@Mindavi thanks for doing this, I'd like to run it on my system. Where would I go to learn more about doing that? Also, I'd like to run it against the zfsUstable version. Any tips?

[Mindavi]

To run this, clone my pull request (#110873), go to the nixos tests folder and run nix-build zfs-tests.nix. Not sure how to run against zfsUnstable, maybe ask someone on IRC?

[stale[bot]]

I marked this as stale due to inactivity. → More info

[bryanasdev000]

Up, this is good stuff.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[bryanasdev000]

Up, this is good stuff.