Reorganizing the -rp and -ca binary packages

[sraustein]

Now that work on BGPSEC certs is winding down, it's time to think about this business of reorganizing which bits go in which package. See #633 for some of the history.

Original organizing theory was that -rp should not depend on Python, but it's been years since that concept made any sense at all. So we tweaked it to say "OK, can depend on Python, but not on any custom Python libraries or extension modules." That is now looking increasingly silly, as the only alternative is to keep writing hideously complicated C code to do relatively trivial things using the OpenSSL C API when we could just be using the Python API we already wrote. Poster children for this include find-roa-expiration and the new scan-routercerts script I wrote as part of the router certificate support code.

So we had this new theory that we should reorganize existing -rp and -ca packages to add a third -lib package, as many other projects do. Turns out that's not really necessary in our case, as -ca is pretty much locked into depending on -rp at this point, so anything that would go in -lib can just as well go in -rp.

So the question boils down to: what needs to move from -ca to -rp?

• rpki.POW needs to move. This is key.
• All of the pure Python rpki.* library code can move. We could try to be more selective, but there's no real point: library code that's not being run isn't hurting anybody.
• Executable programs that are logically part of -ca should stay in -ca, even though the real code is all in the libraries in many cases. So, OK, it looks a little weird to move all of rpki.rpkid to -rp while leaving the two line executable script that loads rpki.rpkid in -ca, but, really, so what? The thing that matters here is the user experience, and what the user will notice is whether ${sbindir}/rpkid is present or not.
• Package dependencies will need to split up: dependencies needed to build rpki.POW will need to move to -rp; dependencies needed to run -ca tools can and should stay with -ca, so that they're not in the face of a user who just wants to run -rp.
• A bunch of stuff in the source code organization will need to change, but, again, who cares? Nobody but the developers look at the source tree anymore if they can avoid it (more precisely, they never did, they just couldn't avoid it until we had binary packages). So some stuff will move but nobody will care so long as what we install doesn't break.

Useful side effect of all this will be that rpki.autoconf will be moving to -rp, so the kludges to generate executable scripts in -rp will finally go away and all the scripts can use rpki.autoconf.

This will involve enough changes that it will almost certainly require a branch, but the total amount of work is probably not all that large.

Comments, before I launch into this?

Trac ticket #685 component build priority major, owner sra, created by sra on 2014-04-05T19:14:03Z, last modified 2014-04-10T22:56:48Z

[sraustein]

Note that the branch that implements this will create several zillion subversion "tree conflicts" when merged with anything else done between when it forks and when it merges. This doesn't mean we need to freeze anything, just coordinate a bit more tightly than usual, be mindful of what's happening, and check the merge results a bit more carefully than is usually required.

Trac comment by sra on 2014-04-05T19:22:25Z

[sraustein]

In [changeset:"5756"]: {{{

!CommitTicketReference repository="" revision="5756"

Branch, see #685. }}}

Trac comment by sra on 2014-04-05T21:12:52Z

[sraustein]

what is the impact on the footprint and binary dependencies of a running rp instance? that is to say, does this significantly increase resource requirements for running a cache? there is gonna be a jillion caches and few cas.

Trac comment by randy on 2014-04-06T00:46:42Z

[sraustein]

what is the impact on the footprint and binary dependencies of a running rp instance? that is to say, does this significantly increase resource requirements for running a cache? there is gonna be a jillion caches and few cas.

Well, we already require Python itself (rpki-rtr, rcynic-html, etc), so I'm going to leave the base Python numbers out of it.

FreeBSD 8-STABLE with shared libraries (results edited to remove irrelevant scripts that aren't installed):

{{{ $ find * -type f -perm -1 | xargs du -s | sort -nr | awk '{print $2}' | xargs du -hs 2.1M rcynic/rcynic 336k rpkid/rpki/POW/_POW.so 66k rtr-origin/rtr-origin 38k utils/print_roa/print_roa 34k utils/print_rpki_manifest/print_rpki_manifest 28k utils/uri/uri 28k utils/scan_roas/scan_roas 28k utils/find_roa/find_roa 26k utils/hashdir/hashdir 22k rcynic/rcynic-html 16k rpkid/portal-gui/scripts/rpkigui-apache-conf-gen 12k rpkid/rpki-sql-setup 12k rpkid/irbe_cli 10k rpkid/rpki/gui/app/range_list.py 10k rpkid/rpki-confgen 8.0k rcynic/rcynic-svn 6.0k rpkid/portal-gui/scripts/rpkigui-import-routes 4.0k utils/scan_routercerts/scan_routercerts 4.0k rpkid/rpki-start-servers 4.0k rpkid/portal-gui/scripts/rpkigui-query-routes 4.0k rpkid/portal-gui/scripts/rpkigui-check-expired 4.0k rcynic/rcynic-text 4.0k rcynic/rcynic-cron 2.0k rpkid/rpkid 2.0k rpkid/rpkic 2.0k rpkid/rpki-sql-backup 2.0k rpkid/rootd 2.0k rpkid/pubd 2.0k rpkid/portal-gui/scripts/rpkigui-rcynic 2.0k rpkid/portal-gui/scripts/rpki-manage 2.0k rpkid/irdbd 2.0k rcynic/validation_status 2.0k rcynic/make-tal.sh }}}

Note that rcynic is statically linked in this configuration, which gives you some idea of the size of the OpenSSL libraries. _POW.so is smaller on this machine because it can use shared libraries; if it couldn't (eg, current Ubuntu or Debian), size would be more like rcynic.

All of the utils/ programs are C code; they're relatively small here because they're using shared libraries, on Ubuntu or Debian they would again be more like the size of rcynic here.

rtr-origin is a Python script, just a really long one which needs refactoring.

With the new scheme:

• rcynic would be approximately the same (smaller if dynamically linked, but there is no OpenSSL shared library we can use on platforms other than FreeBSD and Fedora, so much of the bloat would remain)
• All of the utils/ C programs would eventually become relatively short Python scripts.
• _POW.so would move from rpki-ca to rpki-rp.
• The change in runtime binary dependencies would be scripts depending on _POW.so instead of C programs linking against libcrypto.{a,so}.

I can run these numbers again on Ubuntu for comparison if you like.

Trac comment by sra on 2014-04-06T01:25:38Z

[sraustein]

The other thing to consider, though, is that the C environment is a much more painful place to be writing code.

Eg, the interesting bit of the new scan_routercerts script is twelve lines of Python code, and took about twenty minutes to write, most of that spent refreshing my memory on what ASN.1 fields I needed to extract and what API incantations I needed to extract and print them).

You do not want to know what the half-written C version of that looked like before I abandoned it, but it had already taken a lot more than twenty minutes and was nowhere near done. It would have worked, eventually, but the bulk of it would have been of the form:

{{{ if ((temp_var_372 = obscure_call_3304(temp_var_223, fee, fie, foe, fum)) == NULL) goto fail; }}}

ending up with a closing block like:

{{{ fail: thing_1_free(temp_var_1); thing_2_free(temp_var_2); thing_3_free(temp_var_3); ... thing_3678_free(temp_var_3678); return result; }}}

Ghu knows I've written enough code like that, but it's not the sort of thing one does casually, which is the point. Any task performed in that kind of environment is a major effort even for somebody who is comfortable with the OpenSSL API.

This is what killed the deeply embedded systems market: developers became more expensive than hardware. I suspect you're going to have a hard time finding a (working) PC so bad that Linux running on it can't run the Python code, even in the third world. I guess if you need to run rpki-rtr on your wristwatch you might have a problem, but it doesn't seem like a very common use case.

Trac comment by sra on 2014-04-06T01:49:03Z

[sraustein]

{{{ $ find * -type f -perm -1 | xargs du -s | sort -nr | awk '{print $2}' | xargs du -hs 2.1M rcynic/rcynic 336k rpkid/rpki/POW/_POW.so 66k rtr-origin/rtr-origin 38k utils/print_roa/print_roa

and smaller and smaller.

i guess that about says it

randy

Trac comment by randy on 2014-04-06T01:56:11Z

[sraustein]

Here are numbers for Ubuntu 12.04. Environmental differences:

• rcynic is not statically compiled, but
• We can't use the system OpenSSL (libcrypto.so), so all binaries are linked against our private libcrypto.a.

{{{ 2.5M openssl/openssl-1.0.1f/apps/openssl 2.0M rpkid/rpki/POW/_POW.so 1.7M rcynic/rcynic 1.5M utils/print_roa/print_roa 1.5M utils/print_rpki_manifest/print_rpki_manifest 1.5M utils/scan_roas/scan_roas 1.5M utils/find_roa/find_roa 1.5M utils/uri/uri 1.2M utils/hashdir/hashdir 68K rtr-origin/rtr-origin 24K rcynic/rcynic-html 12K rpkid/rpki/gui/app/range_list.py 12K rpkid/rpki-sql-setup 12K rpkid/rpki-confgen 12K rpkid/irbe_cli 8.0K rcynic/rcynic-svn 4.0K utils/scan_routercerts/scan_routercerts 4.0K utils/print_roa/strip_roa.sh 4.0K rtr-origin/server.sh 4.0K rpkid/rpkid 4.0K rpkid/rpkic 4.0K rpkid/rpki-start-servers 4.0K rpkid/rpki-sql-backup 4.0K rpkid/rootd 4.0K rpkid/pubd 4.0K rpkid/irdbd 4.0K rcynic/validation_status 4.0K rcynic/rcynic-text 4.0K rcynic/rcynic-cron 4.0K rcynic/make-tal.sh }}}

Summary:

• "hello world" linked against libcrypto.a would be at least 1MB
• Two copies of "hello world" linked against libcrypto.a would take up more space than _POW.so linked against libcrypto.a.
• So just the dumb stuff in utils/ is a bigger hit than _POW.so.

Trac comment by sra on 2014-04-06T05:16:57Z

[sraustein]

• rcynic is not statically compiled

s/compiled/linked/. On FreeBSD, we include -static in the linker flags, to make it easier to run chrooted.

Including -static wouldn't help on Linux, because:

1. Some Linux packages don't include .a libraries at all, and
2. There are a few bits of glibc that load further dynamic libraries (which aren't even known by the loader to be dependencies!) at runtime based on what they find in configuration files.

Thus -static is useless on Linux, so we don't even try (anymore).

Trac comment by sra on 2014-04-06T05:24:13Z

[sraustein]

2.5M openssl/openssl-1.0.1f/apps/openssl 2.0M rpkid/rpki/POW/_POW.so 1.7M rcynic/rcynic 1.5M utils/print_roa/print_roa 1.5M utils/print_rpki_manifest/print_rpki_manifest 1.5M utils/scan_roas/scan_roas 1.5M utils/find_roa/find_roa 1.5M utils/uri/uri 1.2M utils/hashdir/hashdir

but it's really

2.5M openssl/openssl-1.0.1f/apps/openssl 2.0M rpkid/rpki/POW/_POW.so 1.7M rcynic/rcynic

and then you probably run one utility at a time

randy

Trac comment by randy on 2014-04-06T05:58:33Z

[sraustein]

2.0M rpkid/rpki/POW/_POW.so 1.7M rcynic/rcynic

and then you probably run one utility at a time

In general, on the RP side, you only run one OpenSSL-using program at a time in any case. To the extent that you might ever try to run more than one at once, you're better off with Python scripts and _POW.so.

On FreeBSD you might be able to do better by tweaking compilation options for both the RPKI code and /usr/ports/security/openssl, but you'd have to compile both from source (ie, forget about pkg install) to get any benefit.

Trac comment by sra on 2014-04-06T06:17:07Z

[sraustein]

Changes from #674 pulled from trunk to this branch, but tree conflicts created by this branch made that merge more of an adventure than usual. See notes in [5766] if something goes horribly wrong when merging all this back to trunk.

Trac comment by sra on 2014-04-07T22:08:05Z

[sraustein]

On 04/07/2014 03:08 PM, Trac Ticket System wrote:

Changes from #674 pulled from trunk to this branch, but tree conflicts created by this branch made that merge more of an adventure than usual. See notes in [5766] if something goes horribly wrong when merging all this back to trunk.

I'll hold off committing anything to trunk until this branch gets merged.

Trac comment by melkins on 2014-04-07T22:16:25Z

[sraustein]

I'll hold off committing anything to trunk until this branch gets merged.

If you really need to commit something, just do it and we'll clean up the mess afterwards. [5766] wasn't particularly hard, I'm mostly just being paranoid about merge and history tracking.

But I hope to have this sorted within the next day or two. At this point everything works in the reorganized tree except for installation and binary packages, so most of what's left is just autoconf and Makefile tweaks.

Trac comment by sra on 2014-04-07T22:28:40Z

[sraustein]

As of [5773], installation for the new reorganized packages seems to work on Ubuntu. Not yet seriously tested, and still on branch, not merged to trunk yet.

Note: rpki-ca depends on rpki-rp, as it should, but this means that the package tools want to upgrade rpki-rp before upgrading rpki-ca. Ordinarily this is fine, but in this case it causes a transient conflict, since the python libraries are present in both the old rpki-ca and the new rpki-rp. There may be some fancy way of fixing this, but the simple answer is:

{{{ apt-get remove rpki-?? apt-get install rpki-?? }}}

Need to confirm that this doesn't do anything horrible to config files or SQL. Note absence of apt-get purge, this is deliberate.

If anybody who knows more about the Debian package tools than I sees a cleaner way to deal with this, please speak up.

FreeBSD port not hacked yet, that comes next after Debian and Ubuntu have been debugged.

Trac comment by sra on 2014-04-09T04:49:49Z

[sraustein]

In [changeset:"5777"]: {{{

!CommitTicketReference repository="" revision="5777"

Upgrade find-roa-expiration to supported tool, per user request. See #633, #685. }}}

Trac comment by sra on 2014-04-10T02:11:22Z

[sraustein]

FreeBSD port and pkg working. Essentially the same caveat as on Ubuntu: moving files from one package to the other creates a transient apparently conflict, and while there may be some "correct" way to do this in-place, it's not obvious. So: {{{ pkg delete rpki-* rpki-portmaster }}}

(or whatever works with the tools you use, point is you have to deinstall temporarily for this upgrade to work).

This will require mail to rpki@ list.

If anybody feels like reviewing the new source tree layout, now's the time. Otherwise I will merge this to trunk Real Soon Now, like tomorrow.

Trac comment by sra on 2014-04-10T02:19:12Z

[sraustein]

Michael pointed out that the Debian tools must have some way of dealing with files moving from one package to another. He was right. [5779] is a one line change to the control file which says it's OK for current versions of rpki-rp to stomp on some of the files that used to be in rpki-ca. With that change, binary upgrade on Ubutnu appears to be seamless.

Doubt there's anything we can do on FreeBSD.

Currently we're testing source installation ("make install" the old fashioned way).

Trac comment by sra on 2014-04-10T17:44:42Z

[sraustein]

In [changeset:"5784"]: {{{

!CommitTicketReference repository="" revision="5784"

Merge tk685 branch back to trunk. This completes the move of the rpki libraries and rpki.POW module from the rpki-ca package to the rpki-rp package. Closes #685, closes #633. }}}

Trac comment by sra on 2014-04-10T22:56:48Z

[sraustein]

Closed with resolution fixed