search

coreos / rpm-ostree

⚛📦 Hybrid image/package system with atomic upgrades and package layering
https://coreos.github.io/rpm-ostree
Other
858 stars 194 forks source link

Test data for composes #90

Closed cgwalters closed 7 years ago

cgwalters commented 9 years ago

Ideally, we would have something for "make check"/InstalledTests that runs though a basic tree compose. I was a bit surprised to see that at least yum doesn't appear to have any test repo metadata or packages.

librepo does: https://github.com/Tojaj/librepo/tree/master/tests/test_data and it also has tests that use it. A technical reason this is nice for librepo is that it can run as non-root.

But unfortunately 'compose tree' would bomb out with yum if we're running "make check" as non-root (as we're constrained to do by RPM %check in a Koji/Brew environment).

Ideas:

cgwalters commented 9 years ago

Also, the Docker approach would give us uid 0 in a container, which would allow us to write much more interesting tests that more fully exercise the "running as uid 0" parts of ostree/yum while still providing system saftey. Unfortunately, these tests couldn't be run in Brew, but that'd be OK, there are other systems that could support running them.

james-antill commented 9 years ago

Yeh, on the yum side we have unit tests for the internal stuff which runs at %check time. But we just rely on developer testing, mock/etc. and shipping it in rawhide for external testing. QA has some infrastructure that allows them to easily create test packages and repos., and they use that to create small test cases with real data (I think the infrastructure code is public, but it'd still be annoying to create 100+ packages I think).

Not sure if you know about: http://fakeroot.alioth.debian.org/ ... but that is in Fedora.

giuseppe commented 9 years ago

neither fakeroot or yocto's pseudo project are able to emulate all that is needed by "rpm-ostree compose tree".

I've started working on an LD_PRELOAD library that should shadow all the calls that require root from rpm-ostree, it doesn't seem like a trivial task, there are many cases that must be handled. I am not yet convinced that this is the correct way though, it seems very fragile.

cgwalters commented 9 years ago

pesudo is fairly actively maintained at least. What system calls failed? If it's some of the container bits, we could just turn that off (though obviously, the more we do this, the less we're testing the real system).

cgwalters commented 9 years ago

Options I see:

Run as non-root

Installed root tests: sudo make rootcheck

This is like https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests but as root. We have to do a sudo make install over the running system, but there's a test suite that operates using the installed binaries, but doesn't break the host. This is similar to what https://github.com/projectatomic/atomic/blob/master/test.sh is doing.

Inject code into a local Docker container

A variant of the above where we do make rpm using a local spec file, then inject it into a Docker (or other) container. Or just make install DESTDIR=, then copy that over into a container. There are a lot of advantages to this - e.g. we could support testing a rawhide build from a RHEL7 host, etc.

Inject code into a VM

Like the above, but with VMs. This is what Cockpit (and other projects I'm sure) do: https://github.com/cockpit-project/cockpit/blob/332c0e4813a541304014603c87cb1e394a3fe307/test/testvm.py#L127

Where the VM approach would be useful for us is that ostree obviously wants to change the kernel configuration. However, this type of testing I think just doesn't fit very well into the make check model. It's best to have it live externally IMO, and make check remains unprivileged unit tests.

giuseppe commented 9 years ago

For example pseudo misses emulation unshare.

I've continued adding wrappers of libc functions, until I realized I was ending up with a clone of pseudo. So I reverted to using pseudo and add a new library for the missing pieces, it is much less code to maintain now:

https://github.com/giuseppe/rpm-ostree/tree/giuseppe/test-compose

TODO:

1) Move the emulation of unshare to pseudo. 2) Properly support execle in pseudo, so we can test the post-process phase as well. It seems that it doesn't honor a chroot. 3) RPM-ostree calls directly the clone syscall unless mount fails with EINVAL. This probably should be still done, or we need to wrap it in a way to allow us to emulate it through LD_PRELOAD.

cgwalters commented 9 years ago

I think we could skip emulation of some of the container syscalls safely enough. Emulating clone is a bit trickier...maybe we could just degrade it to fork without any container features?

Can we depend on pseudo as a package at some point instead of a submodule? A submodule is ok for now though.

giuseppe commented 9 years ago

IMHO, it is safer if the version used in the tests is more loose than the version used by rpm-ostree and we keep using clone and emulate it as best as we can as the "make check" version of compose is not much more than a smoke test. I plan to remove the hack in the fork wrapper once the exec* wrappers in pseudo honors the chroot settings (I have a patch here) and thus we will be able to run depmod and dracut within the rootfs and do the no-op there instead of the fork wrapper itself.

giuseppe commented 9 years ago

after thinking more about it, I think such a complex test as running a compose should rather be done using an installed tests and run as root. We should introduce "make check tests" only for unit testing as we do with ostree. What do you think?

giuseppe commented 9 years ago

I pushed some other commits to:

https://github.com/giuseppe/rpm-ostree/tree/giuseppe/test-compose

I reverted the usage of pseudo, as it looks too fragile environment for any serious testing.

I've moved the compose test under installed tests, and expect they are executed as root user. There is one warning coming from glibc inotify implementation that makes tests fail on Fedora 21/22, this could be fixed by changing the G_DEBUG setting in libtest.sh (or overloading it in the compose tests) but since the tests run without issues on Rawhide, I left it unchanged.

cgwalters commented 9 years ago

Root installed tests sound OK to me...it hasn't been defined in the spec, but that doesn't block us from doing it =) Maybe something like Type=headless-root or something?

cgwalters commented 9 years ago

Nothing blocks us though from doing the sudo make rootcheck option, right? That might be easier than root-installed tests.

giuseppe commented 9 years ago

Yes, that can be done as well. Would you drop installed tests completely or provide both functionalities?

giuseppe commented 9 years ago

it doesn't look like the test repository will work smoothly in the same way from installed tests and when we run tests locally with make check.

Do you think it makes sense to take rid of the installed tests completely and rewrite the tests to be run only locally and maintain only one version?

I will add a check to check if the tests requiring root permissions run with UID=0 or skip them otherwise.

giuseppe commented 9 years ago

I've sent a pull request for the compose test. I think we can make the cleanup later, in case we decide to drop installed tests and use only make check (and we should probably make this uniformly with ostree)

cgwalters commented 7 years ago

This was done.