search

google / capsicum-test

Test suite for Capsicum
BSD 2-Clause "Simplified" License
46 stars 30 forks source link

Fix casper compilation on FreeBSD #26

Closed ngie-eign closed 5 years ago

ngie-eign commented 5 years ago

This makes some inroads towards fixing compilation on OSX, however, there are issues with the ad hoc copy of libnv, et al, that cause linker failures on OSX. This doesn't occur on FreeBSD because it comes with libnv, et al, since 10.x/11.x. This needs to be fixed up, since the tests should either be consistently pulling libraries from the base system, or not doing so (and in turn, pulling it from another appropriate source repository).

This pull request requires #25 in order to function.

daviddrysdale commented 5 years ago

Hmm, I wonder if it might be better to just ignore the casper/ code altogether – what's your overall aim?

IIRC the casper/ code here is derived from the upstream FreeBSD version, as of an old version (~2014, r266830), where I was experimenting with a rough port to check that it worked with a version of Linux that had Capsicum ported to it. I'd consider the upstream FreeBSD code to be the main source, and so updating a 4-year old experimental port doesn't seem sensible.

So if your aim is to have a cross-platform set of Casper tools, I'd suggest starting from the current FreeBSD version and making that version build and work on other platforms.

OTOH if you're just fiddling with casper/ because of the gTest upgrade, then I'd suggest doing the least amount of work under casper/ – maybe even ignore it altogether?

(And apologies – if I'd replied sooner on #25 I might have saved you some work)

ngie-eign commented 5 years ago

@daviddrysdale: hmm... I thought the tests in this repo were newer than the ones in the FreeBSD repo.

What I would like to do is replace the ATF (https://github.com/jmmv/atf) test code with googletest and have the tests be available for all supported platforms. The former is less portable compared to the latter.

daviddrysdale commented 5 years ago

Ah, the tests in casper/tests/ might not be present in the upstream FreeBSD code – I seem to remember writing some tests from scratch to check the experimental port was working. But I don't think there was much to them, so for Casper-related work it's still probably best to start from current FreeBSD.

By the way, where are the ATF-based tests you mention? I don't think I've seen those.

Also, I'm still not clear about your overall goal – are you working on Casper, or the Capsicum test suite, or both?

ngie-eign commented 5 years ago

@daviddrysdale: Ah, the tests in casper/tests/ might not be present in the upstream FreeBSD code – I seem to remember writing some tests from scratch to check the experimental port was working. But I don't think there was much to them, so for Casper-related work it's still probably best to start from current FreeBSD.

Gotcha!

Well (sigh), I was wrong about there being a number of tests in the FreeBSD tree :D.. We have some tests, but not too terribly many. They're also all in TAP (well, shoehorned into it because they use assert(3) liberally to catch errors, which breaks kyua runs), not ATF :(.

Tests integrated into Jenkins (CI)

Also, I'm still not clear about your overall goal – are you working on Casper, or the Capsicum test suite, or both?

tl;dr: Both.

I was directed to this repo originally by @emaste (another committer with the FreeBSD project; a co-mentor of mine), and he and I have a vested interest in getting more test coverage for system components. In particular, there are a number of gaps on the Capsicum side that we would like to cover. Fixing issues with this repo as they crop up, allows us (actually: me :P) to make progress towards achieving that goal.

Having this project integrate into FreeBSD directly is less of a goal, since the build framework is so different (but it would be nice to have it build on FreeBSD so we can do side-by-side comparisons when upgrading the test suite). It's more important to have this integrate into Linux and OSX CI, so potential porting issues with FreeBSD can be caught earlier (FreeBSD can be closely approximated to Linux in some regards and OSX in other regards when it comes to building and testing the project).

My gut feeling about this is that FreeBSD (and Linux) should be contributing to a common project with all of the involved components (libcapsicum, libcasper, libnv, libpjdlog, etc), so there aren't multiple divergent forks, separate parties can contribute to capsicum and casper as a whole, and the tests can be used on all platforms where capsicum and casper are supported.

I acknowledge that this is a very lofty goal and I would need to involve all appropriate parties in order to make progress in this effort.

-Enji (ngie@FreeBSD.org)

CC: @oshogbo, @pjd, @rwatson

emaste commented 5 years ago

We have some tests, but not too terribly many. They're also all in TAP (well, shoehorned into it), not ATF :(.

At least some of these tests ended up directly in FreeBSD as the author wasn't interested in agreeing to the CLA required on the upstream repo. If we import capsicum-test into contrib we could at a minimum migrate them there.

ngie-eign commented 5 years ago

@emaste: At least some of these tests ended up directly in FreeBSD as the author wasn't interested in agreeing to the CLA required on the upstream repo. If we import capsicum-test into contrib we could at a minimum migrate them there.

Yeah, that's been a problem for FreeBSD developers. The Google CLA is a barrier that they don't want to deal with--or can't maybe, because lawyers/bureaucracy, which was the case when I used to work for Dell EMC/Isilon.

daviddrysdale commented 5 years ago

Backing up for a second, this repo is a bit of a mishmash of anything Capsicum-related that runs in userspace, so it helps to be clear about which of the 3 main parts we're talking about:

  1. The top-level directory has the test suite for Capsicum core functionality.
  2. casper/ is my old experiment with Casper, as discussed above.
  3. libcaprights/ holds userspace libraries for the Capsicum Linux port.

Top-level test suite

The main test suite is only expected to run on Capsicum-enabled OSes, which is essentially just FreeBSD and a patched version of the Linux kernel. When last I checked, it built and ran on both1 with only a few minor test failures on FreeBSD.

So the top-level test suite might well be a good candidate for integrating with FreeBSD testing, but isn't relevant to OSX (no Capsicum) and wouldn't work with stock Linux (no Capsicum without the patched kernel) so any CI integration would be FreeBSD-only.

Codebase status-wise, AFAIK this repo is the master copy of the test suite and so is a good place to start from (but maybe the CLA thing means there are some extra tests out there too). However, I'm not actively developing anything on the Linux side so it may make sense to fork and treat a new location as the master.

Casper directory

I like the idea of an independent, CI-tested repo for the various Casper-related tools and utilities, and also the idea that the Casper code could be more OS-agnostic and so build and have tests on multiple platforms (FreeBSD, Linux, OSX). However, this repo probably isn't the place to start from nor to hold the master copy of the code.

tl;dr: don't start from here.

Libcaprights directory

This is entirely Linux-Capsicum specific, so can probably be ignored in this discussion.

1: The test suite currently has a hack involving the different priorities of GNUmakefile and makefile so that it works with both gmake on FreeBSD and make on Linux without any more complicated autotools-like setup.

daviddrysdale commented 5 years ago

Assuming all that makes sense, what I think it would boil down to as concrete suggestions:

emaste commented 5 years ago

I'm happy to fork it into either the freebsdfoundation or freebsd github orgs (although I don't have permission for the latter - should check into that).

emaste commented 5 years ago

We could fork this into either the FreeBSD Foundation GitHub org (for which I have appropriate permissions) or preferably the FreeBSD project's org (for which I don't), but @bapt will fork it there if that is what we want.

ngie-eign commented 5 years ago

@daviddrysdale: I'm abandoning this PR because of the discussion we've had here and also #25 . I will work on your recommended steps for pushing #25 forward.

Is there a way that we could work with the Google lawyers on the capsicum tests or just reassign the project to the FreeBSD project's care to avoid having to deal with the Google CLA and make sure that everything is in the clear for future contributions and such?