Set up Cirrus CI

[JeremyRand]

As everyone has probably heard by now, Travis CI is shutting down its gratis service for freedom software projects on December 31. Bitcoin Core is moving to Cirrus CI. (The Bitcoin Core devs have identified security concerns about GitHub Actions and Microsoft Azure Pipelines, so those are not an option.)

This would be a good opportunity to move the Namecoin-specific repos to Cirrus CI, and set up additional useful CI stuff that we didn't have before with Travis. For example, integration tests of the ncdns NSIS installer and integration tests of DNS and TLS interoperability with mainstream browsers would help us identify problems a lot faster. Also, build artifact uploading would make it a lot easier for beta testers to try out our code (since it wouldn't require me to build things myself just for them to test it). It would also be really cool to run rbm on Cirrus, so that Cirrus can act as a co-signer for our reproducible builds. (The Tor guys are curious if we can pull this off; it would benefit Tor too if we can do it.) And of course switching to a modern static analyzer (gometalinter has been deprecated for ages) would help assure that we're not pushing messy code.

[JeremyRand]

@domob1812 Given that the overhead imposed by not even having Travis available is interfering with my ability to efficiently do https://github.com/namecoin/ncdns/issues/124#issuecomment-748893997 , and I don't want to charge money for time that I'm spending inefficiently on manual testing, can we authorize funding 1 week of development via Handshake funds on setting up Cirrus on the Namecoin-specific repos? Seems like that would be much better-spent time than having me write code with much worse efficiency than I usually can do. I think a week is likely to cover enough Cirrus stuff that I can get back to work on https://github.com/namecoin/ncdns/issues/124#issuecomment-748893997 without impairment (and probably with a significant productivity boost, e.g. there's a good chance I can implement some CI for the ncdns NSIS installer, which currently has no CI whatsoever).

[domob1812]

If you think that you can get it all ready in a week, I think that's certainly worth paying for.

[JeremyRand]

@domob1812 Week 1 has been completed. This involved adding almost all of the Cirrus functionality we needed for the certinject, ncdns, and ncdns-nsis repos (it caught some nice ncdns-nsis bugs, which I fixed), and was (as predicted) sufficient to get a new Windows ncdns release out, which included system-wide TLS negative overrides. It also included some significant experiments with using Cirrus for the ncdns-repro repo. These experiments demonstrated significant promise, but I didn't have time in Week 1 to get things building completely for that repo, which is why the new ncdns release does not have non-Windows binaries available.

@domob1812 Is it okay to pay out for Week 1?

If a Week 2 is approved, I would like to focus on ncdns-repro (and its upstream, tor-browser-build). I've found some highly useful info that I think should allow us to build ncdns (and Tor Browser) reproducibly on Cirrus, thus allowing Cirrus infrastructure to serve as an automated co-signer for reproducible ncdns and Tor Browser binaries. This will be of interest to both the Namecoin and Tor communities (so it will help reinforce Namecoin's reputation as a good neighbor that contributes to Tor). I also think I can refactor some of ncdns-repro's code to minimize the amount of developer overhead it imposes on us (in particular, I think I can crater the number of merge conflicts that happen when upstream Tor Browser changes things).

@domob1812 Is it okay to allocate Handshake funds for Week 2?

[domob1812]

Sounds good to me. After the proposed week 2, will there be anything left to do for Cirrus, or will all that you need and can think of be done then?

[JeremyRand]

Sounds good to me. After the proposed week 2, will there be anything left to do for Cirrus, or will all that you need and can think of be done then?

@domob1812 There are some minor things like IRC notifications that I need to do (Cirrus makes this harder than Travis did, but it should be doable), and there are some less commonly touched repos (e.g. tlsrestrictnss) that need Cirrus added. I will try to do the former if there's spare time in Week 2. I will probably not spend time on the latter until I find myself interacting with one of those repos (a lot of those repos are mostly inactive for a reason, e.g. because they exist only for backwards-compatibility with stuff that we no longer recommend; the main exceptions are the ncp11 and its dependencies, which are not deprecated, but which I don't want to spend time on until we get Encaya released).

[JeremyRand]

If a Week 2 is approved, I would like to focus on ncdns-repro (and its upstream, tor-browser-build). I've found some highly useful info that I think should allow us to build ncdns (and Tor Browser) reproducibly on Cirrus

This is now working for building ncdns, ncp11, and ncprop279 reproducibly in rbm on Cirrus. I haven't yet enabled artifact uploads, but that should be easy. Tor Browser is going to be a little bit harder, because there are more rbm projects in Tor Browser, and I will have to patch a few of them that don't exist in Namecoin (particularly Rust and maybe Firefox) to properly handle my checkpoint implementation.

thus allowing Cirrus infrastructure to serve as an automated co-signer for reproducible ncdns and Tor Browser binaries.

Once artifact uploads are working (see above), I should be able to trivially implement co-signing via Signify or GPG.

This will be of interest to both the Namecoin and Tor communities (so it will help reinforce Namecoin's reputation as a good neighbor that contributes to Tor).

I just talked to Gus from Tor, and I've been approved to give a presentation about this at Tor Demo Day on Wednesday (Feb 24).

I also think I can refactor some of ncdns-repro's code to minimize the amount of developer overhead it imposes on us (in particular, I think I can crater the number of merge conflicts that happen when upstream Tor Browser changes things).

A lot of this is now complete. The vast majority of patches we make to rbm.conf are now handled automatically by a patching script. There are a few minor things in rbm.conf that still are patched in manually (mainly related to checkpoints), which I will need to automate. And I should also automate the patches to Makefile (which should be quite easy). I also want to make Cirrus automatically submit PR's when dependency versions are bumped; that will be nontrivial but probably not super hard either.

There are some minor things like IRC notifications that I need to do (Cirrus makes this harder than Travis did, but it should be doable)

I've coded up a Matrix bot called First Intelligence Agency of the People (name is a joke reference to the historical CIA/KGB IRC bots and to a quote from a Chelsea Manning legal document) that will notify us when Cirrus builds fail. It hasn't gotten any real-world testing yet (no Cirrus failures since I turned on the bot) but it should get some testing in the next couple days.

there are some less commonly touched repos (e.g. tlsrestrictnss) that need Cirrus added.

I added Cirrus to the generate_nmc_cert repo, since I was touching that repo anyway in order to add Encaya cert support to it.