Testing with Travis CI

[runningnak3d]

After discussing on Reddit (post from @paradigm below), I am opening this issue to track the progress of that implementation, and track any additional tests that should be run.

Some example things which we could test, if it helps:

make check

make GPGID=

Hijacking various traditional distros

Commands like brl enable, brl repair, and brl apply which manipulate bind mounts, FUSE mounts, symlinks, directories, and normal files, and change config file contents.

Shell functions

Bedrock is almost all side effects, mocking probably isn't worthwhile

C functions

Bedrock is almost all side effects, mocking probably isn't worthwhile

If testing brl fetch feasible that'd be particularly important as that regularly breaks as upstream changes things, but I figure travis probably doesn't grant us internet access as that would be abusable. If it is feasible, could we have it run regularly (e.g. once a day or once a week) and notify us when it fails rather than (just) on PRs so we could be notified of breaking upstream changes?

Don't feel constrained by this list if you have other ideas.

If we can also run tests locally as well I think there's value there.

My personal preference leans towards simplicity/minimalism where feasible and I use stuff similar to this in other projects I work on, but those tend to be simpler projects as well with things like far fewer side effects and if you feel like there's real need for something more substantial I'm perfectly amenable.

I'm also really excited to see the SIGTERM/SysV issue resolved. It bothers me as a matter of principle that we have issues there.

[cptpcrd]

FYI: Travis recently changed their pricing model. The TL;DR of it is that free users now receive a fixed amount of credits for build minutes; when they are exhausted, you must buy new credits to continue using Travis. Previously (I believe) it was a monthly usage limit.

As a result, any effort to implement this would probably want to use GitHub Actions (which, actually, is arguably better) instead of Travis. I toyed around with both in a branch on my fork, but I never got around to PRing it.

---

Note: One of the reasons I didn't PR the changes I put together is that it isn't actually possible to properly test hijacking/fetching -- you really need to reboot the VM to test that, and GitHub Actions/Travis don't seem to support rebooting.

The only way I can think of to test this properly is maybe spinning up a VM (yes, a VM inside a VM!) with QEMU or something. I'm willing to investigate further if there's interest in getting this working.

[bobbbay]

For reference, a GitHub bot would be quite cool as well.

[cptpcrd]

For reference, a GitHub bot would be quite cool as well.

Hmm... that could be interesting; could you expand further? What do you think a bot would be most helpful with?

[paradigm]

I've been thinking about this quite a bit while working on plans for 0.8 Naga for the last few months. Naga is most likely going to end up being a substantial rewrite to fix some architectural design issues that didn't age well with 0.7 Poki. I plan to make the code base much more test friendly from day one with that effort.

Naga is probably going to be more C heavy than Poki. I'm currently working on a small C unit test framework along with build system changes to do things like code coverage checks and valgrind checks against the tests, along with aggressive static analysis tool usage. I actually finished the framework itself; I'm exercising it against another toy project to see if there's anything I need to improve with it now before I start relying on it.

Once that's done, the very next thing on my plate is to write a small utility to do something like chroot into a Naga system, but in a Bedrock-aware fashion. It probably won't work with Poki, sadly; changes will have to made to play nicely with it. This will be useful to reintroduce Bedrock's manual install option as well as debugging and emergency fixing. For our purposes here, though, this will let us do integration/e2e/system/whatever tests cleanly, including things like brl fetch, without the need for VMs. I'm not sure about testing hijacking, though everything short of hijacking is probably good enough. Maybe we could test hijacking the github/travis/whatever environment itself? :P I plan on making a test framework around this bedrock-chroot utility.

Once all that's ready, I'll make a new branch - probably something like 0.8alpha - and develop against it. From there I'll leave it to you fine folks to figure out how to integrate it with github or travis or whatever is best. Provided it's free-of-charge, gives us enough resources to do what we need to do, and is easy to switch off of, I don't have adequate background on these things to have a reason to be picky.

[ghost]

Maybe we could test hijacking the github/travis/whatever environment itself?

Hijacking against CI probably wouldn't work, unless there's some CI that I don't know about that has restart support. Maybe if you wanted to test hijacking while the system is on and you didn't want to worry about the hijacking after reboot, then it could work, maybe. You could also maybe find a CI service that gives you a direct VM to play with, but I don't know of anything like that.

On the other hand:

The only way I can think of to test this properly is maybe spinning up a VM (yes, a VM inside a VM!) with QEMU or something.

Never thought about that, although that sounds like it would be very messy. I once tried running a VM in a docker container, it was not pretty. But, if you could get it right, that might not be the worst idea, as it solves the problem below.

...with github or travis or whatever is best...gives us enough resources to do what we need to do

Unless you have a self-hosted runner, GitHub Actions doesn't have multi-architecture support, so that might not be good. The only multi-arch CI I know of is Travis, but it only supports amd64, ppc64le, s390x, and arm64, so I'm actually not too confident in those two. If anyone knows of a CI service that has support for the architectures that Bedrock supports, let me know.

...easy to switch off of...

Ease to switch off of probably won't be too big of a deal, it's just learning the potentially different CI script format.

Naga is probably going to be more C heavy than Poki

I actually like this change, until now, it kind of felt like bedrock was just a collection of shell scripts, which you can't really test. With this, it would be a lot easier.

...to write a small utility to do something like chroot into a Naga system

I'm very excited about this! I've been annoyed that there isn't a good "chroot" utility for bedrock, and I'm glad it's coming!

[paradigm]

Naga is probably going to be more C heavy than Poki

I actually like this change, until now, it kind of felt like bedrock was just a collection of shell scripts, which you can't really test. With this, it would be a lot easier.

May be worth pointing out that, by SLOC, Poki is much more C than shell already:

$ sloccount bedrocklinux-userland | grep '^[a-z]*:'
ansic:         3640 (81.87%)
sh:             806 (18.13%)

However, the C isn't as user-facing as the shell. This was intentional, as shell is easier for end-users to tweak. For Naga, at least, I plan on retaining some shell specifically because I know those are places users like to poke around in, e.g. brl fetch scripts.

The fact you didn't notice the C bits is something of a complement. Users use Bedrock not for its own sake, but to get access to other things. Ideally, Bedrock proper should stay out of the way and be unnoticed. As Futurama put it, when you do things right, people won't be sure you've done anything at all..

[bobbbay]

I'm late to the conversation, but can contribute:

Hmm... that could be interesting; could you expand further? What do you think a bot would be most helpful with?

With a custom GitHub bot, you could have pretty much anything done. I once dabbled a bit with GitHub bots, pretty easy work. I got one to run all the scripts in my .ctrlrc/ directory every time someone made a PR. Yeah, any shebangable script can be run, even hijacking. Pretty cool stuff, I have an existing repo and work on it already, from a week ago (I think, time flies).

Let me know if you'd like details.

I think Valgrind checks are great, and hijacking tests are a must - heck, I'm almost surprised we don't have CI like that here 😉

Cheers!