search

dragonresearch / rpki.net

Dragon Research Labs rpki.net RPKI toolkit
54 stars 26 forks source link

work0: st00pid user can not do the portupgrade install #582

Open sraustein opened 11 years ago

sraustein commented 11 years ago

{{{ work0.psg.com:/usr/src# sh rpki-portupgrade.sh ==> Downloading rpki-rp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3104 100 3104 0 0 9192 0 --:--:-- --:--:-- --:--:-- 33376 x rpki-rp/ x rpki-rp/pkg-install x rpki-rp/pkg-plist x rpki-rp/Makefile x rpki-rp/pkg-deinstall x rpki-rp/pkg-descr x rpki-rp/pkg-message x rpki-rp/pkg-upgrade x rpki-rp/distinfo ==> Downloading rpki-ca % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4750 100 4750 0 0 74496 0 --:--:-- --:--:-- --:--:-- 76612 x rpki-ca/ x rpki-ca/files/ x rpki-ca/Makefile x rpki-ca/pkg-descr x rpki-ca/pkg-plist x rpki-ca/pkg-install x rpki-ca/pkg-deinstall x rpki-ca/pkg-upgrade x rpki-ca/distinfo x rpki-ca/files/rpki-ca.in ==> Fixing up port Makefiles and whacking RPKI ports into portsdb [Updating the portsdb format:bdb_btree in /usr/ports ... - 24053 port entries found .........1000.........2000.........3000.........4000.........5000.........6000.........7000.........8000.........9000.........10000.........11000.........12000.........13000.........14000.........15000.........16000.........17000.........18000.........19000.........20000.........21000.........22000.........23000.........24000 ..... done] ==> Running portupgrade for rpki-rp ---> Session started at: Mon, 01 Jul 2013 00:17:56 +0000 [Updating the pkgdb format:bdb_btree in /var/db/pkg ... - 348 packages found (-0 +2) .. done] * None has been installed or upgraded. ---> Session ended at: Mon, 01 Jul 2013 00:17:57 +0000 (consumed 00:00:00) ==> Running portupgrade for rpki-ca ---> Session started at: Mon, 01 Jul 2013 00:17:57 +0000 * None has been installed or upgraded. ---> Session ended at: Mon, 01 Jul 2013 00:17:57 +0000 (consumed 00:00:00) }}}

Trac ticket #569 component build priority minor, owner sra, created by randy on 2013-07-01T00:19:38Z, last modified 2013-11-24T00:56:38Z

sraustein commented 11 years ago

Interesting that the RPKI port directories on work0 seem to be owned by buildbot. Something odd on work0 or that just happens to be the uname associated with the UID that gets stuffed in the tar file by the equally robotic process that builds the tarballs in the first place?

Try running script with sh -x so we can see what's happening?

Trac comment by sra on 2013-07-02T02:42:56Z

sraustein commented 11 years ago

{{{ work0.psg.com:/usr/src# sh -x rpki-portupgrade.sh

Trac comment by randy on 2013-07-02T03:48:04Z

sraustein commented 10 years ago

I don't know what's up with this one. rpki-portupgrade has always been more fragile than rpki-portmaster, portupgrade is just like that.

pkgtools.conf on work0 contains what looks like the right magic goop and appears to have last been edited about four minutes before this ticket was filed, so I'm guessing that the right magic goop was in place at the time.

We've disabled a couple of ports-system whoopie cushions since that time, but those are both things that would have caused the ports system itself to fail, not just portupgrade, and they would not have failed silently, so I don't think they're relevant.

Is this still happening, or did it go away as mysteriously as it started?

Trac comment by sra on 2013-11-22T23:05:08Z

sraustein commented 10 years ago

you can try the upgrade or i can

Trac comment by randy on 2013-11-22T23:06:22Z

sraustein commented 10 years ago

you can try the upgrade or i can

I'll try it. Ghu only knows what will happen, that machine has more personality than a '68 Camaro.

Trac comment by sra on 2013-11-22T23:11:25Z

sraustein commented 10 years ago

i am not fond dealing with the consequences of years of adding (and deleting) services on that server.

Trac comment by randy on 2013-11-22T23:13:01Z

sraustein commented 10 years ago

ports_glob doesn't believe that the rpki/* ports exist. Comment in pkgtools.conf suggests that fixing this requires pkgdb -Ufu, not just pkgdb -fu. Trying that now.

If this is correct, it probably renders the whole concept of rpki-portupdate useless. pkgdb -Ufu takes forever to run, as it requires running about 26,000 separate make commands (literally -- one per port, and there are a lot of ports). Seems unlikely that anyone will want to sit through this often.

Trac comment by sra on 2013-11-22T23:33:18Z

sraustein commented 10 years ago

Note: Even if pkgdb -Ufu "fixes" the problem, there is still something we don't understand going on here. This script has worked on other machines without heroic measures. What's different about work0 (other than years and mileage...)?

Trac comment by sra on 2013-11-22T23:40:22Z

sraustein commented 10 years ago

actually, if the darn thing will run, i do not care if it takes a day. i do have other things to work on.

Trac comment by randy on 2013-11-22T23:40:25Z

sraustein commented 10 years ago

actually, if the darn thing will run, i do not care if it takes a day.

You don't care if it takes a day to run every time you try to upgrade?

Also note that, even ignoring the RPKI code, the current state of the pkgdb is not happy. pkgdb -Ff offered to deinstall half a dozen packages as a first step towards resolving its woes (I said "no").

Trac comment by sra on 2013-11-22T23:46:50Z

sraustein commented 10 years ago

ports_glob still doesn't admit that the rpki ports exist. Something else is wrong, no idea what.

Trac comment by sra on 2013-11-23T00:25:43Z

sraustein commented 10 years ago

poor old work0

Trac comment by randy on 2013-11-23T00:40:39Z

sraustein commented 10 years ago

The only thing I can think of left to try is: {{{ rm -f /var/db/pkg/pkgdb.db pkgdb -Ffu }}}

but:

  1. I don't think ports_glob (aka portsdb) even looks at pkgdb.db, so I have no reason to think this would work.
  2. Whacking pkgdb.db is almost always more exciting than one really wanted.
  3. Some of the packages that I know pkgdb is going to ask me about are things I know nothing about.

Trac comment by sra on 2013-11-23T01:00:55Z

sraustein commented 10 years ago

so you want me to do it, in the hope that i will remember more things than you?

Trac comment by randy on 2013-11-23T01:06:25Z

sraustein commented 10 years ago

so you want me to do it, in the hope that i will remember more things than you?

Well, you're more likely to remember what you're doing with, eg, shoutcast, since I never knew.

But I think I'm giving up on getting this particular script to work on this particular machine, at least for now. Bottomless pit.

Trac comment by sra on 2013-11-23T03:03:11Z

sraustein commented 10 years ago

shoutcast is heavily used by ballet etc. and source no longer supported. need to move to soe linux host.

Trac comment by randy on 2013-11-23T04:35:37Z

sraustein commented 10 years ago

ok, i ran {{{ rm -f /var/db/pkg/pkgdb.db pkgdb -Ffu }}} rebuilt a few ports where the dependencies had moved

{{{ work0.psg.com:/usr/src# sh rpki-portupgrade.sh ==> Downloading rpki-rp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3328 100 3328 0 0 14902 0 --:--:-- --:--:-- --:--:-- 14923 x rpki-rp/ x rpki-rp/files/ x rpki-rp/pkg-plist x rpki-rp/Makefile x rpki-rp/pkg-deinstall x rpki-rp/pkg-descr x rpki-rp/pkg-message x rpki-rp/pkg-upgrade x rpki-rp/distinfo x rpki-rp/files/pkg-install.in ==> Downloading rpki-ca % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4880 100 4880 0 0 223k 0 --:--:-- --:--:-- --:--:-- 226k x rpki-ca/ x rpki-ca/files/ x rpki-ca/Makefile x rpki-ca/pkg-descr x rpki-ca/pkg-plist x rpki-ca/pkg-install x rpki-ca/pkg-deinstall x rpki-ca/pkg-upgrade x rpki-ca/distinfo x rpki-ca/files/rpki-ca.in ==> Fixing up port Makefiles and whacking RPKI ports into portsdb [Updating the portsdb format:bdb_btree in /usr/ports ... - 24318 port entries found .........1000.........2000.........3000.........4000.........5000.........6000.........7000.........8000.........9000.........10000.........11000.........12000.........13000.........14000.........15000.........16000.........17000.........18000.........19000.........20000.........21000.........22000.........23000.........24000... ..... done] ==> Running portupgrade for rpki-rp ---> Session started at: Sat, 23 Nov 2013 08:31:50 +0000 * None has been installed or upgraded. ---> Session ended at: Sat, 23 Nov 2013 08:31:50 +0000 (consumed 00:00:00) ==> Running portupgrade for rpki-ca ---> Session started at: Sat, 23 Nov 2013 08:31:51 +0000 * None has been installed or upgraded. ---> Session ended at: Sat, 23 Nov 2013 08:31:51 +0000 (consumed 00:00:00) }}}

Trac comment by randy on 2013-11-23T08:32:40Z

sraustein commented 10 years ago

It's not just work0. Same failure on a VM I'd had shut down and been ignoring for a few months. First step on that machine (not work0) is portsnap, so that's running now, then will take another look.

Trac comment by sra on 2013-11-23T18:40:53Z

sraustein commented 10 years ago

Minor progress:

  1. ports_glob does know about the rpki ports, I was just invoking it incorrectly. {{{ work0.psg.com:/root# ports_glob rpki* rpki/rpki-rp rpki/rpki-ca }}}
  2. I found a minor bug in rpki-portupgrade.sh which was causing it to write the wrong thing in /usr/ports/rpki/Makefile. As far as I can tell, this has no real effect except when running portsdb -Ufu, but it's fixed now in any case.

Now that I know that portsdb does know about the rpki ports, suspicion shifts to pkgdb and portupgrade itself. No particular reason to believe that portsdb -Ufu would help now. Tried pkgdb -fu on test VM, didn't help. Will try a few more variations before putting this back on the shelf again.

At this point I suspect that something in portupgrade itself changed during an upgrade, since this script used to work and we didn't change anything that would have made it stop working. Guess I could look at changes to the portupgrade port since I committed the original working version of the script.

Trac comment by sra on 2013-11-24T00:20:31Z

sraustein commented 10 years ago

pkgdb -Ff did find one duplicated origin upstream of rpki-rp (gio-blah-blah merged into glib-blah, whine whine whine) but fixing that didn't help. Deleting /var/db/pkg/*.db and /usr/ports/*.db didn't help.

Ports themselves are fine, they work if invoked directly or via portmaster. It's just portupgrade.

Trac comment by sra on 2013-11-24T00:33:08Z

sraustein commented 10 years ago

It's just portupgrade.

how gawddam depressing

Trac comment by randy on 2013-11-24T00:56:38Z