Depending on updated/enhanced versions of standard packages

[sraustein]

New ticket, since this isn't about Kimura-san's backtrace anymore. See #418 for context.

I think it may be better just to set up our own apt repository with newer versions of required software. As a test, I rebuilt python-django and python-django-south from the Quantal release and set up a repository:

deb http://sigpipe.org/rpki rpki main

I'm really tempted to say that if we're going to set up our own apt repository, we should use that to get RFC-3779-enabled OpenSSL libraries too, so we can move the binary packaging in a direction that the Debian and Ubuntu maintainers might accept. Not sure what we should do to the package version number in our forked package if we do that to make it all come out right.

Not sure we want to get into this in the next few days. The rpki-ca.postinst hack to run pip install works and may be simpler for the user (no need to fiddle apt repository configuration). Unless somebody feels strongly about this, I will probably just keep the pip hack for next week's hackathon.

Then again, if we did set up our own apt repository, our users could use apt-get instead of dpkg, which might outweigh the pain of configuring the apt repository and work out as a net gain.

I think the apt-repository thing is probably the right direction in the medium run (long run of course is to get this into distros and not our problem). Perhaps document how to do that, and advise me on what to do with the OpenSSL package version number. :)

Trac ticket #423 component rpkid priority minor, owner sra, created by sra on 2013-02-22T03:27:34Z, last modified 2016-08-05T17:28:50Z

[sraustein]

For now I've generated updated packages with the pip hack. If I can get to the point where I believe we can have a private APT repository ready for use and relatively seemless for the user in time for the hackathon, I'm happy to go with that, but what with me substitute teaching for Randy tomorrow I don't want to bet on us having huge chunks of time available to work on this.

I still have a couple of minor things to do to the rpki-rp packages based on Wednesday's RP hackathon, mostly automating the inetd/xinetd setup glop. I'm tempted to use faucet instead of inetd on FreeBSD, so we can avoid the fragile process of patching /etc/services and /etc/inetd.conf (this isn't an issue with xinetd, we just drop our own file into /etc/xinetd.d, no patching required).

Trac comment by sra on 2013-02-22T04:13:45Z

[sraustein]

It looks like the "pip install" hack installs in /usr/local, which sort of makes sense under the Debian/Ubuntu rules but is a bit surprising. Haven't tested yet to see whether it causes problems, nobody's reported any but I don't know how extensively anybody has tested yet.

I think we do want to go towards the private APT solution. I will google, but Michael, if you have a good reference, please pass it along.

Trac comment by sra on 2013-02-28T04:02:03Z

[sraustein]

Articles found so far, which detail far too many alternative ways of doing this (penguins), but seem to agree that the sane option is reprepro.

http://wiki.debian.org/HowToSetupADebianRepository http://www.debian-administration.org/articles/286 http://blog.jonliv.es/2011/04/26/creating-your-own-signed-apt-repository-and-debian-packages/

Trac comment by sra on 2013-02-28T05:58:49Z

[sraustein]

Debian packaging hacks which may be relevant to the OpenSSL problem:

http://www.debian-administration.org/articles/20 http://www.debian-administration.org/articles/118

Trac comment by sra on 2013-02-28T06:15:35Z

[sraustein]

reprepro is what I used in my test, and I agree with your conclusion that it is what should be used.

I didn't get around to signing the packages I generated, but here's what I did to rebuild the deb packages http://sigpipe.org/rpki/rebuild.html

Trac comment by melkins on 2013-02-28T17:00:59Z

[sraustein]

In theory, we can set up a single 64-bit Ubuntu VM and have it build both i386 and amd64 packages, for both Debian and Ubuntu (yes, it sez here that one can build for Debian on an Ubuntu machine). We're not at this level of automation yet, but if and when we get there it will probably be with pbuilder or cowbuilder. References:

http://www.kirya.net/articles/build-i386-packages-on-amd64/ https://wiki.ubuntu.com/PbuilderHowto

Note: this is all still the same toolchain, just further and further layers of encapsulation and automation. debuild is a wrapper around dpkg-buildpackage, pbuilder is a wrapper around dbuild, cowbuild is a wrapper around pbuilder.

General advice seems to be that if we're serious about building packages for others to use we should be at least at the pbuilder level. cowbuilder is just an efficiency hack around pbuilder.

Trac comment by sra on 2013-03-05T16:41:53Z

[sraustein]

uh, while i am following this discussion and googling to try to understand a bit, i have the more urgent issue of being able to do the install on the mac mini for the next workshop and making esnet successful at the end of this week (if michael gets the free time he hopes).

Trac comment by randy on 2013-03-05T23:34:18Z

[sraustein]

There is a method to the madness.

[5090] completely broke our ability to generate binary packages for Ubuntu. While we could attempt to kludge around the mess yet again, our current belief is that the right fix involves setting up our own APT repository with acceptable versions of the packages on which we depend. Which is what this ticket is about.

The rest of this (pbuilder, etc) is just notes on future work to be done in response to comments received verbally over the last few weeks ("No 64-bit? Sniff.", "No Debian? Sniff.", etc). There's a fair amount of overlap in the solutions to all of these issues, so while I'm currently focused on the roadblock, it seemed worthwhile to spend a few hours figuring out the tools and landmines.

What I've got at the moment is a 64-bit Ubuntu VM (under VirtualBox, but using a disk format we can transfer to VMWare later) on which I'm installing the tools necessary to do this stuff. Eventually I expect to have automatic package builds for Debian and Ubuntu running here under cron, but at the moment I'm just making sure I don't do anything so dumb that I'll have to tear this all down and start over.

Trac comment by sra on 2013-03-06T02:41:25Z

[sraustein]

We could revert that change for now. It just means that the rpki-manage helper script won't be usable. It is merely a convenience wrapper around django-admin.py that sets $PYTHONPATH and $DJANGO_SETTINGS_MODULE environment.

Trac comment by melkins on 2013-03-06T02:49:42Z

[sraustein]

I probably need to press on with the stuff on this ticket in any case, but reverting [5090] would take this ticket off Randy's critical path.

Trac comment by sra on 2013-03-06T02:58:05Z

[sraustein]

In [changeset:5109]: {{{

!CommitTicketReference repository="" revision="5109"

revert [5090]

see #423 }}}

Trac comment by melkins on 2013-03-06T03:12:56Z

[sraustein]

I probably need to press on with the stuff on this ticket in any case, but reverting [5090] would take this ticket off Randy's critical path.

does this imply that the packages at

http://download.rpki.net/Ubuntu_Packages/

now will work and i should retry installs on my mini?

Trac comment by randy on 2013-03-06T04:51:20Z

[sraustein]

rpki-manage will still be broken (see https://trac.rpki.net/ticket/431) but you can hand edit the script if you care to use it.

Trac comment by melkins on 2013-03-06T05:03:28Z

[sraustein]

my memory is that rpki-manage was broken because configure broke in the package build. has that been fixed?

and how do i fix the script?

Trac comment by randy on 2013-03-06T05:06:10Z

[sraustein]

does this imply that the packages at

 http://download.rpki.net/Ubuntu_Packages/

now will work and i should retry installs on my mini?

Now that I've manually updated the Ubuntu package, yes.

Ubuntu package updated to 0.5109. FreeBSD package will follow but isn't there yet (needed to do a portsnap/portmaster cycle first).

Trac comment by sra on 2013-03-06T05:08:00Z

[sraustein]

my memory is that rpki-manage was broken because configure broke in the package build. has that been fixed?

no, the rpki-manage generated by the deb package build is still broken.

and how do i fix the script?

the last line of the script should be:

django-admin $*

it may be django-admin.py, so double check. i can't recall what it gets named when django is installed via pip (the ubuntu package calls it django-admin).

Trac comment by melkins on 2013-03-06T05:10:02Z

[sraustein]

thanks. trying ubuntu installs now.

Trac comment by randy on 2013-03-06T05:10:32Z

[sraustein]

cache0:/usr/local/src# dpkg -i rpki-rp_0.5109_i386.deb Selecting previously unselected package rpki-rp. (Reading database ... 52802 files and directories currently installed.) Unpacking rpki-rp (from rpki-rp_0.5109_i386.deb) ... dpkg: dependency problems prevent configuration of rpki-rp: rpki-rp depends on rrdtool; however: Package rrdtool is not installed. rpki-rp depends on xinetd; however: Package xinetd is not installed. dpkg: error processing rpki-rp (--install): dependency problems - leaving unconfigured Errors were encountered while processing: rpki-rp

Trac comment by randy on 2013-03-06T05:12:06Z

[sraustein]

Selecting previously unselected package rpki-rp.

Hint: you have not installed this before, thus probably have not satisfied dependencies. Fix for this is APT repository (original subject of this ticket).

dpkg: dependency problems prevent configuration of rpki-rp: rpki-rp depends on rrdtool; however: Package rrdtool is not installed. rpki-rp depends on xinetd; however: Package xinetd is not installed.

So it's telling you what dependencies it needs. For now, you have to install these yourself with apt-get.

Trac comment by sra on 2013-03-06T06:35:15Z

[sraustein]

sra is working on building an appliance do the builds, but in the meantime i got curious about building the deb packages myself. this repo allows you to use apt-get to install the rpki packages with automatic dependencies:

http://sigpipe.org/rpki/readme.html

Trac comment by melkins on 2013-03-06T23:31:20Z

[sraustein]

can i apt-get them on top of the installs i have done already?

Trac comment by randy on 2013-03-06T23:32:25Z

[sraustein]

i think the only issue may be that since sra's deb packages used pip to install django and south into /usr/local, they may conflict with the python-django and python-django-south standard packages that get installed with my deb packages.

you can remove the packages installed by pip:

pip uninstall django
pip uninstall south

if you run into problems.

Trac comment by melkins on 2013-03-06T23:36:01Z

[sraustein]

and will i need to do all configuration again?

Trac comment by randy on 2013-03-07T00:04:20Z

[sraustein]

probably best to let sra answer. i have not tested what happens on upgrade, only for new deployments.

Trac comment by melkins on 2013-03-07T00:07:00Z

[sraustein]

61 wget -q -O - http://sigpipe.org/rpki/pubkey.asc | apt-key add - 62 wget -q -O /etc/apt/sources.list.d/rpki.list http://sigpipe.org/rpki/rpki.list 63 apt-get update 64 apt-get install rpki-rp rpki-ca 65 e /etc/rpki.conf 66 rpki-sql-setup 67 rpkic initialize 68 initctl start rpki-ca 69 initctl restart rpki-ca 70 ls 71 rpkic configure_parent workshop.xml 72 rpkic configure_publication_client workshop.rgnet.repository-request.xml 73 rpkic configure_repository workshop.repository-response.xml

ca0.vmini.rg.net:/usr/local/src# rpki-manage migrate --list DatabaseError: (1146, "Table 'irdbd.south_migrationhistory' doesn't exist")

Trac comment by randy on 2013-03-07T00:25:20Z

[sraustein]

you need to run

rpki-manage syncdb

first

Trac comment by melkins on 2013-03-07T00:32:09Z

[sraustein]

ok, and made it through config and restarted apache. https gets

Error 102 (net::ERR_CONNECTION_REFUSED): The server refused the
connection

though http tells me to use https

nothing in error log

shall i wipe the vm and install from scratch?

Trac comment by randy on 2013-03-07T01:02:29Z

[sraustein]

my guess is you need to do this:

a2ensite default-ssl service apache2 reload

Trac comment by melkins on 2013-03-07T01:06:08Z

[sraustein]

a2ensite default-ssl service apache2 reload

nope

Trac comment by randy on 2013-03-07T01:07:09Z

[sraustein]

also note that you need to put the "Include ..." to include our apache.conf into default-ssl and not into default.

Trac comment by melkins on 2013-03-07T01:07:45Z

[sraustein]

also note that you need to put the "Include ..." to include our apache.conf into default-ssl and not into default.

Include ...?

where is this in the docco?

Trac comment by randy on 2013-03-07T01:09:06Z

[sraustein]

that error is saying that apache is not listening on port 443. make sure that you see /etc/apache2/sites-enabled/default-ssl

Trac comment by melkins on 2013-03-07T01:09:21Z

[sraustein]

On Thu, Mar 07, 2013 at 01:09:06AM -0000, Trac Ticket System wrote:

423: Depending on updated/enhanced versions of standard packages

--------------------------+------------------- Reporter: sra | Owner: sra Type: enhancement | Status: new Priority: minor | Component: rpkid Resolution: | Keywords: Blocked By: | Blocking: --------------------------+-------------------

Comment (by randy):

also note that you need to put the "Include ..." to include our apache.conf into default-ssl and not into default.

Include ...?

where is this in the docco?

dunno, this is new territory as i didn't make those packages. but its the same basic princple as documented in https://trac.rpki.net/wiki/doc/RPKI/CA/UI/GUI#ConfigureApache

Trac comment by melkins on 2013-03-07T01:10:45Z

[sraustein]

also note that you need to put the "Include ..." to include our apache.conf into default-ssl and not into default. where is this in the docco? https://trac.rpki.net/wiki/doc/RPKI/CA/UI/GUI#ConfigureApache

do not see putting something into apache.conf or default-ssl

Trac comment by randy on 2013-03-07T01:13:02Z

[sraustein]

that error is saying that apache is not listening on port 443. make sure that you see /etc/apache2/sites-enabled/default-ssl

does not exist

Trac comment by randy on 2013-03-07T01:13:24Z

[sraustein]

mod-ssl doesn't get enabled by default, you also need to do this:

a2enmod ssl service apache2 restart

Trac comment by melkins on 2013-03-07T01:13:28Z

[sraustein]

a2enmod ssl service apache2 restart

yay! i get web page!

can you please put all this cruft into the docco?

Trac comment by randy on 2013-03-07T01:15:12Z

[sraustein]

https://trac.rpki.net/wiki/doc/RPKI/CA/UI/GUI#Ubuntu

Trac comment by melkins on 2013-03-07T01:19:59Z

[sraustein]

fyi, just updated that, forgot to install mod_wsgi as well

Trac comment by melkins on 2013-03-07T01:24:36Z

[sraustein]

i have deleted the vm and will build from scratch. i have to know this will work from scratch.

can there please be one page of destructions i (and other idiots) can just follow?

Trac comment by randy on 2013-03-07T01:27:23Z

[sraustein]

probably not possible due to differences in host OS. this is what packaging hopes to solve.

Trac comment by melkins on 2013-03-07T01:28:54Z

[sraustein]

i meant for ubuntu. separate page for freebsd, of course. but i will run workshop on ubuntu and users will install RP and maybe CA

Trac comment by randy on 2013-03-07T01:32:32Z

[sraustein]

In [changeset:5118]: {{{

!CommitTicketReference repository="" revision="5118"

Add "Section: net" header to debian/control; fixes #449. Make both RPKI packages depend on apache2; fixes #445. Get rid of "pip install" hack and change back to proper depends on Django and South to work properly with project APT repository; see #423. mod_ssl ("a2enmod ssl") and mod_wsgi not handled yet, get those tomorrow. }}}

Trac comment by sra on 2013-03-07T07:20:36Z

[sraustein]

sweet!

Trac comment by randy on 2013-03-07T07:27:23Z

[sraustein]

as it turns out, mod_ssl is included in the apache2 package, it just isn't enable by default (debian/rpki-ca.postinst should run "a2enmod ssl")

libapache2-mod-wsgi is the package we want for mod_wsgi

Trac comment by melkins on 2013-03-07T16:58:19Z

[sraustein]

mod_ssl and mod_wsgi added to rpki-ca in [5123].

Trac comment by sra on 2013-03-07T20:05:43Z

[sraustein]

In [changeset:5124]: {{{

!CommitTicketReference repository="" revision="5124"

Whack OpenSSL configuration to pull its notion of the CPU from autoconf rather than deducing this on its own, so that we can use pbuilder and pbuilder-dist to build both 32-bit and 64-bit packages on the same 64-bit Ubuntu machine. See #423. }}}

Trac comment by sra on 2013-03-08T04:32:56Z

[sraustein]

OK, it looks like I may have a working pbuilder-dist + reprepro configuration which builds 32 and 64 bit images for Ubuntu 12.04 LTS. In theory, it can also build other distributions, including Debian 6.0.6 ("Squeeze") but that's a lower priority.

Public appearance is very similar to Michael's test repository (unsurprising, as I took his recommendation on software and will be stealing his instructions as well). Back end is (OK, will be, RSN) a cron job on a VM (currently running under VirtualBox on an old laptop in my house, but we should move it to one of Randy's VMware servers). Basic cycle for cron job is pretty simple:

• svn update
• update / regenerate any Ubuntu specific glorp (eg, debian/changelog)
• run pbuilder-dist for every distribution and architecture we want
• import results into reprepro
• rsync resulting reprepro repository to web server

Last step may be a bit surprising, but was deliberate, I want the build machine to be separate from the public distribution machine, as they have different security and availability issues.

There were a few minor tricky bits in setting up the pbuilder environments, and a few subtle bugs in our code which I had to fix before any of this would work, but at this point it appears to generate usable packages.

I've upgraded another VM using apt-get with this repository to see if the most recent generated images do anything useful. Assuming nothing horrible shows up overnight, I'll post information on how to use this tomorrow and set it up as a cron script.

At some point we may want to switch from pbuilder-dist to cowbuilder-dist, but that's purely an efficiency trade off (disk space for time) and I haven't played with cowbuilder enough yet to know how to set it up to use our private repository for dependencies (may turn out to be identical to pbuilder, just don't know yet).

Still to do is cleanup of how I'm handling the debian/ subdirectory (Michael has been very polite about this, but what I'm doing now is just plain disgusting, and needs to be cleaned up). This only affects people trying to build packages, not the end users, so it's been a lower priority than getting the APT repository running.

Trac comment by sra on 2013-03-08T06:47:37Z

[sraustein]

we should move it to one of Randy's VMware servers

what do you want on the server? ubuntu 64, and you build the 32-bit with -m32? how much disk and ram? any particular hostname?

Trac comment by randy on 2013-03-08T07:05:54Z

[sraustein]

= How to use the rpki.net APT repository =

• Add the GPG public key for this repository (optional, but APT will whine unless you do this):

{{{

!sh

wget -q -O - http://download.rpki.net/APT/apt-gpg-key.asc | sudo apt-key add - }}}

Public key block:

{{{ pub 2048R/EFFB1DA8 2013-03-06 Key fingerprint = A362 A1AA 9284 BFEA 6B84 693E A2C4 27B5 EFFB 1DA8 uid rpki.net Debian/Ubuntu package signing key debkey@rpki.net sub 2048R/BBE5888C 2013-03-06 }}}

• Configure APT to use this repository:

{{{

!sh

sudo wget -q -O /etc/apt/sources.list.d/rpki.list http://download.rpki.net/APT/rpki.ubuntu.list }}}

• Update available packages:

{{{

!sh

sudo apt-get update }}}

• Install the software:

{{{

!sh

sudo apt-get install rpki-rp rpki-ca }}}

Trac comment by sra on 2013-03-09T04:52:19Z