No covering certificate for <rpki.rpkid.roa_obj {labuser03

[sraustein]

continual whining

Apr 18 01:50:28 work0 rpkid[94998]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping Apr 18 01:58:36 work0 dhcpd: uid lease 147.28.0.68 for client 00:0c:29:d9:34:44 is duplicate on 147.28.0.0/24 Apr 18 02:00:31 work0 rpkid[94998]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping Apr 18 02:02:31 work0 rpkid[94998]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping Apr 18 02:04:31 work0 rpkid[94998]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping

but, when i go to the gui, https://demo.rpki.net/rpki/, as rgnet, i see the labuser03 child looks the same as all the others.

Trac ticket #517 component rpkid priority minor, owner sra, created by randy on 2013-04-19T02:56:48Z, last modified 2013-05-07T04:33:32Z

[sraustein]

but, when i go to the gui, https://demo.rpki.net/rpki/, as rgnet, i see the labuser03 child looks the same as all the others.

Try clicking the refresh button and see if the cert disappears. I have not figured out a good way to automatically refresh that list, because querying rpkid takes a not-insignificant amount of time, and its annoying to the user when it won't change most of the time.

Trac comment by melkins on 2013-04-19T04:13:35Z

[sraustein]

i chose identity labuser03 refresh made it go away chose identity rgnet went to assign 98.128.3.0 to labuser03

"Overlap with previous allocation to this child"

Trac comment by randy on 2013-04-19T04:22:02Z

[sraustein]

On 04/18/2013 09:22 PM, Trac Ticket System wrote:

517: No covering certificate for <rpki.rpkid.roa_obj {labuser03

---------------------+------------------- Reporter: randy | Owner: sra Type: defect | Status: new Priority: minor | Component: rpkid Resolution: | Keywords: Blocked By: | Blocking: ---------------------+-------------------

Comment (by randy):

i chose identity labuser03 refresh made it go away chose identity rgnet went to assign 98.128.3.0 to labuser03

"Overlap with previous allocation to this child"

The IRDB does contain the record indicating that the prefix should be allocated to labuser03, but for some reason labuser03 doesn't see any cert from its parent:

{{{ In [1]: from rpki.irdb.models import ChildNet

In [6]: o=ChildNet.objects.filter(childhandle='labuser03', childissuer__handle='rgnet')[0]

In [7]: o.start_ip, o.end_ip Out[7]: (u'98.128.3.0', u'98.128.3.255')

}}} {{{ work0.psg.com:/var/log# irbe_cli list_received_resources --self_handle=labuser03

/var/log# irbe_cli list_received_resources --self_handle=labuser03

<?xml version='1.0' encoding='us-ascii'?>

}}}

Trac comment by melkins on 2013-04-19T20:59:03Z

[sraustein]

The certificate is in the publication directory, and valid according to an RP on a different host.

{{{ work0.psg.com:/usr/local/var/rpki/conf/rgnet/publication/rgnet/711# openssl x509 -inform der -noout -text -in SaP9A-gNaD374et42u0x1B2uAlQ.cer Certificate: Data: Version: 3 (0x2) Serial Number: 259 (0x103) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=5D88D34E81E8A5899EFA1970814C94C98443FD54 Validity Not Before: Apr 17 06:41:40 2013 GMT Not After : Apr 17 05:00:03 2014 GMT Subject: CN=49A3FD03E80D683DFBE1EB78DAED31D41DAE0254 Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (2048 bit) Modulus (2048 bit): 00:e8:53:df:38:24:0c:de:1e:bc:6b:26:a0:f4:02: 69:27:81:1d:e1:b6:53:cf:69:33:84:43:ad:33:76: ae:ed:4b:4e:f2:57:08:31:09:0a:05:64:25:4f:ff: 45:d4:cf:7b:29:af:42:59:3b:2a:de:09:7d:4b:d9: 72:de:d2:6c:a0:02:35:22:6a:e1:35:67:1c:01:b9: 8d:c9:4b:2c:b8:34:d9:b0:d3:57:76:6f:f0:42:8b: f9:7c:63:91:fa:2b:44:de:e3:73:92:0f:a2:8e:7e: 9d:98:d1:6f:60:da:24:2c:32:4e:16:59:5e:d7:9e: 38:1e:50:cb:bd:44:97:97:93:9d:a1:eb:89:03:21: a7:5b:02:62:21:2b:dc:9f:cf:e0:1a:e7:23:bf:6a: da:9e:b3:07:09:4a:c7:4d:a6:a8:44:a8:af:7d:1b: cc:9b:0f:57:60:f5:42:09:55:f1:86:5a:fc:1d:fb: 02:17:70:53:32:5f:18:37:dc:42:07:ab:0a:dc:38: 26:3f:52:bd:7e:11:76:2d:16:b4:dd:e6:61:35:31: cc:07:9d:a6:8b:87:12:d4:16:b8:94:37:e8:43:bf: ee:df:e6:14:ea:ba:64:10:63:5c:6c:07:dc:53:e5: 78:73:19:3f:0a:3b:a7:ef:00:bf:0e:be:da:0a:af: f0:e9 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 49:A3:FD:03:E8:0D:68:3D:FB:E1:EB:78:DA:ED:31:D4:1D:AE:02:54 X509v3 Authority Key Identifier:

keyid:5D:88:D3:4E:81:E8:A5:89:9E:FA:19:70:81:4C:94:C9:84:43:FD:54

        X509v3 Certificate Policies: critical
            Policy: 1.3.6.1.5.5.7.14.2

        X509v3 CRL Distribution Points:

URI:rsync://rgnet.rpki.net/rpki/rgnet/711/XYjTToHopYme-hlwgUyUyYRD_VQ.crl

        Authority Information Access:
            CA Issuers - URI:rsync://repo0.rpki.net/rpki/root/iana/arin/6/XYjTToHopYme-hlwgUyUyYRD_VQ.cer

        X509v3 Basic Constraints: critical
            CA:TRUE
        X509v3 Key Usage: critical
            Certificate Sign, CRL Sign
        Subject Information Access:
            CA Repository - URI:rsync://rgnet.rpki.net/rpki/rgnet/labuser03/714/
            1.3.6.1.5.5.7.48.10 - URI:rsync://rgnet.rpki.net/rpki/rgnet/labuser03/714/SaP9A-gNaD374et42u0x1B2uAlQ.mft

        sbgp-ipAddrBlock: critical
            IPv4:
              98.128.3.0/24

}}}

Trac comment by melkins on 2013-04-19T21:05:46Z

[sraustein]

well, it's a comfort that it can confuse you too :)

Trac comment by randy on 2013-04-19T21:07:45Z

[sraustein]

labuser03 doesn't seem to be querying its parent: {{{ Apr 19 21:03:24 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.PollParentTask {labuser03}> Apr 19 21:03:24 work0 rpkid[94998]: Self labuser03[7] polling parents Apr 19 21:03:24 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.UpdateChildrenTask {labuser03}> Apr 19 21:03:24 work0 rpkid[94998]: Self labuser03[7] updating children Apr 19 21:03:24 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.UpdateROAsTask {labuser03}> Apr 19 21:03:24 work0 rpkid[94998]: Self labuser03[7] updating ROAs Apr 19 21:03:24 work0 rpkid[94998]: Issuing query for ROA requests Apr 19 21:03:24 work0 rpkid[94998]: Received response to query for ROA requests Apr 19 21:03:24 work0 rpkid[94998]: Couldn't find existing ROA, created <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> Apr 19 21:03:24 work0 rpkid[94998]: <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> doesn't exist, generating Apr 19 21:03:24 work0 rpkid[94998]: Searching for new ca_detail for ROA <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> Apr 19 21:03:24 work0 rpkid[94998]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping Apr 19 21:03:24 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.UpdateGhostbustersTask {labuser03}> Apr 19 21:03:24 work0 rpkid[94998]: Self labuser03[7] updating Ghostbuster records Apr 19 21:03:24 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.RegenerateCRLsAndManifestsTask {labuser03}> Apr 19 21:03:24 work0 rpkid[94998]: Self labuser03[7] regenerating CRLs and manifests Apr 19 21:03:24 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.CheckFailedPublication {labuser03}> }}}

The IRDB does have a Parent object for labuser03:

{{{ In [1]: from rpki.irdb.models import Parent

In [2]: parent = Parent.objects.get(issuer__handle='labuser03')

In [3]: parent Out[3]: }}}

But rpkid seems to have forgotten about that parent:

{{{ work0.psg.com:/usr/local/var/rpki/conf/rgnet/publication/rgnet/711# irbe_cli parent --action=list --self_handle=labuser03

<?xml version='1.0' encoding='us-ascii'?>

<?xml version='1.0' encoding='us-ascii'?>

}}}

Trac comment by melkins on 2013-04-19T21:09:55Z

[sraustein]

This seems to have been happening for quite some time, as the oldest logfile I could find also shows that labuser03 forgot its parent, so no hints there.

I am inclined to say try to run "rpkic -i labuser03 synchronize" but maybe Rob will want to diagnose the core problem before we change anything.

{{{ -rw-r--r-- 1 root wheel 4490312 Apr 13 00:00 rpkid.log.6.gz }}}

{{{ Apr 12 00:01:37 work0 rpkid[32343]: Running task <rpki.rpkid_tasks.PollParentTask {labuser03}> Apr 12 00:01:37 work0 rpkid[32343]: Self labuser03[7] polling parents Apr 12 00:01:37 work0 rpkid[32343]: Running task <rpki.rpkid_tasks.UpdateChildrenTask {labuser03}> Apr 12 00:01:37 work0 rpkid[32343]: Self labuser03[7] updating children Apr 12 00:01:37 work0 rpkid[32343]: Running task <rpki.rpkid_tasks.UpdateROAsTask {labuser03}> Apr 12 00:01:37 work0 rpkid[32343]: Self labuser03[7] updating ROAs Apr 12 00:01:37 work0 rpkid[32343]: Issuing query for ROA requests Apr 12 00:01:37 work0 rpkid[32343]: Received response to query for ROA requests Apr 12 00:01:37 work0 rpkid[32343]: Couldn't find existing ROA, created <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> Apr 12 00:01:37 work0 rpkid[32343]: <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> doesn't exist, generating Apr 12 00:01:37 work0 rpkid[32343]: Searching for new ca_detail for ROA <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> Apr 12 00:01:37 work0 rpkid[32343]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping Apr 12 00:01:37 work0 rpkid[32343]: Running task <rpki.rpkid_tasks.UpdateGhostbustersTask {labuser03}> Apr 12 00:01:37 work0 rpkid[32343]: Self labuser03[7] updating Ghostbuster records Apr 12 00:01:37 work0 rpkid[32343]: Running task <rpki.rpkid_tasks.RegenerateCRLsAndManifestsTask {labuser03}> Apr 12 00:01:37 work0 rpkid[32343]: Self labuser03[7] regenerating CRLs and manifests Apr 12 00:01:37 work0 rpkid[32343]: Running task <rpki.rpkid_tasks.CheckFailedPublication {labuser03}> }}}

Trac comment by melkins on 2013-04-19T21:17:20Z

[sraustein]

labuser03 doesn't seem to be querying its parent: ... The IRDB does have a Parent object for labuser03: ... But rpkid seems to have forgotten about that parent:

Fun!

First guess is that somebody redid the setup dance for labuser03 and only got as far as configure_parent. We don't synchronize the IRDB with rpkid after configure_parent, because doing so would produce an immediate failure when rpkid attempted to request a certificate and publish a CRL and manifest, because the child's repository isn't set up yet.

The weakness in this theory is that there's a recent RPKI certificate, which implies that the parent somehow got a valid PKCS#10 from the child, which can't happen if the child isn't polling the parent.

Trac comment by sra on 2013-04-20T23:18:39Z

[sraustein]

so what should i do here?

Trac comment by randy on 2013-04-25T06:48:27Z

[sraustein]

bump

Trac comment by randy on 2013-05-01T18:47:19Z

[sraustein]

As discussed previously, it looks like somebody messed up the parent/child relationship.

Do you want instructions for redoing the setup dance or do you just want it fixed?

Trac comment by sra on 2013-05-01T20:50:55Z

[sraustein]

As discussed previously, it looks like somebody messed up the parent/child relationship.

except that you said

The weakness in this theory is that there's a recent RPKI certificate, which implies that the parent somehow got a valid PKCS#10 from the child, which can't happen if the child isn't polling the parent.

which i assumed, obviously incorrectly, you wanted to examine and debug.

fwiw, i thought we did the same thing for labuser[00-32] or whatever. of course mistakes are possible.

Do you want instructions for redoing the setup dance or do you just want it fixed?

either

Trac comment by randy on 2013-05-01T21:29:19Z

[sraustein]

rpkid database:

{{{ mysql> select parent_id, parent_handle, sia_base, sender_name, recipient_name, self_handle, last_cms_timestamp from parent, self where parent.self_id = self.self_id; +-----------+---------------+--------------------------------------------------+---------------+----------------+---------------+---------------------+ | parent_id | parent_handle | sia_base | sender_name | recipient_name | self_handle | last_cms_timestamp | +-----------+---------------+--------------------------------------------------+---------------+----------------+---------------+---------------------+ | 1 | arin | rsync://rgnet.rpki.net/rpki/rgnet/ | RGNETI-1 | arin | rgnet | 2013-05-06 21:18:07 | | 7 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser02/ | labuser02 | rgnet | labuser02 | 2013-05-06 21:18:09 | | 9 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser04/ | labuser04 | rgnet | labuser04 | 2013-05-06 21:18:09 | | 10 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser05/ | labuser05 | rgnet | labuser05 | 2013-05-06 21:18:09 | | 11 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser06/ | labuser06 | rgnet | labuser06 | 2013-05-06 21:18:09 | | 12 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser07/ | labuser07 | rgnet | labuser07 | 2013-05-06 21:18:09 | | 13 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser08/ | labuser08 | rgnet | labuser08 | 2013-05-06 21:18:10 | | 14 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser09/ | labuser09 | rgnet | labuser09 | 2013-05-06 21:18:10 | | 15 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser10/ | labuser10 | rgnet | labuser10 | 2013-05-06 21:18:10 | | 16 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser11/ | labuser11 | rgnet | labuser11 | 2013-05-06 21:18:10 | | 17 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser12/ | labuser12 | rgnet | labuser12 | 2013-05-06 21:18:10 | | 18 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser13/ | labuser13 | rgnet | labuser13 | 2013-05-06 21:18:10 | | 19 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser14/ | labuser14 | rgnet | labuser14 | 2013-05-06 21:18:10 | | 20 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser15/ | labuser15 | rgnet | labuser15 | 2013-05-06 21:18:11 | | 21 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser16/ | labuser16 | rgnet | labuser16 | 2013-05-06 21:18:11 | | 22 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser17/ | labuser17 | rgnet | labuser17 | 2013-05-06 21:18:11 | | 23 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser18/ | labuser18 | rgnet | labuser18 | 2013-05-06 21:18:11 | | 24 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser19/ | labuser19 | rgnet | labuser19 | 2013-05-06 21:18:11 | | 25 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser20/ | labuser20 | rgnet | labuser20 | 2013-05-06 21:18:12 | | 26 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser21/ | labuser21 | rgnet | labuser21 | 2013-05-06 21:18:12 | | 27 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser22/ | labuser22 | rgnet | labuser22 | 2013-05-06 21:18:12 | | 28 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser23/ | labuser23 | rgnet | labuser23 | 2013-05-06 21:18:12 | | 29 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser24/ | labuser24 | rgnet | labuser24 | 2013-05-06 21:18:12 | | 30 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser25/ | labuser25 | rgnet | labuser25 | 2013-05-06 21:18:12 | | 31 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser26/ | labuser26 | rgnet | labuser26 | 2013-05-06 21:18:13 | | 32 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser27/ | labuser27 | rgnet | labuser27 | 2013-05-06 21:18:13 | | 33 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser28/ | labuser28 | rgnet | labuser28 | 2013-05-06 21:18:13 | | 34 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser29/ | labuser29 | rgnet | labuser29 | 2013-05-06 21:18:13 | | 35 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser30/ | labuser30 | rgnet | labuser30 | 2013-05-06 21:18:13 | | 36 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser31/ | labuser31 | rgnet | labuser31 | 2013-05-06 21:18:13 | | 37 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser32/ | labuser32 | rgnet | labuser32 | 2013-05-06 21:18:13 | | 38 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/scratchmonkey/ | scratchmonkey | rgnet | scratchmonkey | 2013-05-06 21:18:14 | | 39 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/prasantha/ | prasantha | rgnet | prasantha | 2013-05-06 21:18:14 | | 40 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/serpil/ | serpil | rgnet | serpil | 2013-05-06 21:18:14 | | 41 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser00/ | labuser00 | rgnet | labuser00 | 2013-05-06 21:18:14 | | 42 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser01/ | labuser01 | rgnet | labuser01 | 2013-05-06 21:18:08 | +-----------+---------------+--------------------------------------------------+---------------+----------------+---------------+---------------------+ }}}

• So irbe_cli report in previous comment was correct that labuser03 doesn't know its parent.
• parent_id is autoincrement, so this says that parent entries for labuser00 and labuser01 were created after parent entries for labuser02-labuser32. Seems unlikely that we'd do that in a script, although I don't really remember anymore (original labuser setup was a 3am job in a hotel in Hong Kong).

Trac comment by sra on 2013-05-06T21:22:53Z

[sraustein]

rpkic says (obscurely -- completion failure in delete_repository) that labuser03 has no repository configured.

Trac comment by sra on 2013-05-06T21:25:10Z

[sraustein]

Did:

{{{ rpkic> select_identity labuser03

rpkic> delete_publication_client rgnet/labuser03

rpkic> delete_parent rgnet

rpkic> select_identity rgnet

rpkic> delete_child labuser03

rpkic> create_identity labuser03 Wrote /root/labuser03.identity.xml This is the "identity" file you will need to send to your parent

rpkic> select_identity rgnet

rpkic> configure_child labuser03.identity.xml Child calls itself 'labuser03', we call it 'labuser03' Wrote /root/rgnet.labuser03.parent-response.xml Send this file back to the child you just configured

rpkic> select_identity labuser03

rpkic> configure_parent rgnet.labuser03.parent-response.xml Parent calls itself 'rgnet', we call it 'rgnet' Parent calls us 'labuser03' Wrote /root/labuser03.rgnet.repository-request.xml This is the file to send to the repository operator

rpkic> configure_publication_client labuser03.rgnet.repository-request.xml This looks like a referral, checking Client calls itself 'labuser03', we call it 'rgnet/labuser03' Client says its parent handle is 'rgnet' Wrote /root/rgnet.labuser03.repository-response.xml Send this file back to the publication client you just configured

rpkic> configure_repository rgnet.labuser03.repository-response.xml Repository calls us 'rgnet/labuser03' Repository response associated with parent_handle 'rgnet'

rpkic> synchronize }}}

The final "synchronize" was almost certainly unnecessary, the other commands are supposed to synchronize their bits as they go along.

Trac comment by sra on 2013-05-06T21:37:53Z

[sraustein]

labuser03 now sees its parent, but still doesn't get its ROA. Resource configuration oops?

Note "No overlap" message.

{{{ May 6 21:32:12 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.PollParentTask {labuser03}> May 6 21:32:12 work0 rpkid[94998]: Self labuser03[7] polling parents May 6 21:32:12 work0 rpkid[94998]: Sending "list" request to parent rgnet May 6 21:32:12 work0 rpkid[94998]: Sweeping <rpki.left_right.child_elt {rgnet} labuser03> May 6 21:32:12 work0 rpkid[94998]: Serving list query from child labuser03 [sender labuser03, recipient rgnet] May 6 21:32:12 work0 rpkid[94998]: No overlap between received resources and what child labuser03 should get ([ASN: 3130,3927,3970,4128, V4: 67.21.36.0/24,69.166.11.0/24,98.128.0.0/16,147.28.0.0/16,192.83.230.0/24,192.169.0.0/23,198.133.206.0/24,198.180.150.0-198.180.153.255,207.34.0.0/24,216.21.0.0/24,216.21.14.0/2\ 4,216.21.16.0/24,216.151.34.0/24,216.151.36.0/24,216.151.38.0/24,216.151.41.0/24], []) May 6 21:32:12 work0 rpkid[94998]: Sweeping <rpki.left_right.parent_elt {labuser03} rgnet> May 6 21:32:12 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.UpdateChildrenTask {labuser03}> May 6 21:32:12 work0 rpkid[94998]: Self labuser03[7] updating children May 6 21:32:12 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.UpdateROAsTask {labuser03}> May 6 21:32:12 work0 rpkid[94998]: Self labuser03[7] updating ROAs May 6 21:32:12 work0 rpkid[94998]: Issuing query for ROA requests May 6 21:32:12 work0 rpkid[94998]: Received response to query for ROA requests May 6 21:32:12 work0 rpkid[94998]: Couldn't find existing ROA, created <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> May 6 21:32:12 work0 rpkid[94998]: <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> doesn't exist, generating May 6 21:32:12 work0 rpkid[94998]: Searching for new ca_detail for ROA <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24> May 6 21:32:12 work0 rpkid[94998]: No covering certificate for <rpki.rpkid.roa_obj {labuser03} 4128 98.128.3.0/24>, skipping May 6 21:32:12 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.UpdateGhostbustersTask {labuser03}> May 6 21:32:12 work0 rpkid[94998]: Self labuser03[7] updating Ghostbuster records May 6 21:32:13 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.RegenerateCRLsAndManifestsTask {labuser03}> May 6 21:32:13 work0 rpkid[94998]: Self labuser03[7] regenerating CRLs and manifests May 6 21:32:13 work0 rpkid[94998]: Running task <rpki.rpkid_tasks.CheckFailedPublication {labuser03}> }}}

Trac comment by sra on 2013-05-06T21:39:20Z

[sraustein]

Raw SQL confirms that labuser03 now knows its daddy. So we had more than one problem here (I'm shocked).

{{{ mysql> select parent_id, parent_handle, sia_base, sender_name, recipient_name, self_handle, last_cms_timestamp from parent, self where parent.self_id = self.self_id; +-----------+---------------+--------------------------------------------------+---------------+----------------+---------------+---------------------+ | parent_id | parent_handle | sia_base | sender_name | recipient_name | self_handle | last_cms_timestamp | +-----------+---------------+--------------------------------------------------+---------------+----------------+---------------+---------------------+ | 1 | arin | rsync://rgnet.rpki.net/rpki/rgnet/ | RGNETI-1 | arin | rgnet | 2013-05-06 21:42:13 | | 7 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser02/ | labuser02 | rgnet | labuser02 | 2013-05-06 21:42:15 | | 9 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser04/ | labuser04 | rgnet | labuser04 | 2013-05-06 21:42:15 | | 10 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser05/ | labuser05 | rgnet | labuser05 | 2013-05-06 21:42:15 | | 11 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser06/ | labuser06 | rgnet | labuser06 | 2013-05-06 21:42:16 | | 12 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser07/ | labuser07 | rgnet | labuser07 | 2013-05-06 21:42:16 | | 13 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser08/ | labuser08 | rgnet | labuser08 | 2013-05-06 21:42:16 | | 14 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser09/ | labuser09 | rgnet | labuser09 | 2013-05-06 21:42:16 | | 15 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser10/ | labuser10 | rgnet | labuser10 | 2013-05-06 21:42:16 | | 16 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser11/ | labuser11 | rgnet | labuser11 | 2013-05-06 21:42:16 | | 17 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser12/ | labuser12 | rgnet | labuser12 | 2013-05-06 21:42:16 | | 18 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser13/ | labuser13 | rgnet | labuser13 | 2013-05-06 21:42:17 | | 19 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser14/ | labuser14 | rgnet | labuser14 | 2013-05-06 21:42:17 | | 20 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser15/ | labuser15 | rgnet | labuser15 | 2013-05-06 21:42:17 | | 21 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser16/ | labuser16 | rgnet | labuser16 | 2013-05-06 21:42:17 | | 22 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser17/ | labuser17 | rgnet | labuser17 | 2013-05-06 21:42:17 | | 23 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser18/ | labuser18 | rgnet | labuser18 | 2013-05-06 21:42:17 | | 24 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser19/ | labuser19 | rgnet | labuser19 | 2013-05-06 21:42:18 | | 25 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser20/ | labuser20 | rgnet | labuser20 | 2013-05-06 21:42:18 | | 26 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser21/ | labuser21 | rgnet | labuser21 | 2013-05-06 21:42:18 | | 27 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser22/ | labuser22 | rgnet | labuser22 | 2013-05-06 21:42:18 | | 28 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser23/ | labuser23 | rgnet | labuser23 | 2013-05-06 21:42:18 | | 29 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser24/ | labuser24 | rgnet | labuser24 | 2013-05-06 21:42:18 | | 30 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser25/ | labuser25 | rgnet | labuser25 | 2013-05-06 21:42:18 | | 31 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser26/ | labuser26 | rgnet | labuser26 | 2013-05-06 21:42:19 | | 32 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser27/ | labuser27 | rgnet | labuser27 | 2013-05-06 21:42:19 | | 33 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser28/ | labuser28 | rgnet | labuser28 | 2013-05-06 21:42:19 | | 34 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser29/ | labuser29 | rgnet | labuser29 | 2013-05-06 21:42:19 | | 35 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser30/ | labuser30 | rgnet | labuser30 | 2013-05-06 21:42:19 | | 36 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser31/ | labuser31 | rgnet | labuser31 | 2013-05-06 21:42:19 | | 37 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser32/ | labuser32 | rgnet | labuser32 | 2013-05-06 21:42:19 | | 38 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/scratchmonkey/ | scratchmonkey | rgnet | scratchmonkey | 2013-05-06 21:42:20 | | 39 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/prasantha/ | prasantha | rgnet | prasantha | 2013-05-06 21:42:20 | | 40 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/serpil/ | serpil | rgnet | serpil | 2013-05-06 21:42:20 | | 41 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser00/ | labuser00 | rgnet | labuser00 | 2013-05-06 21:42:20 | | 42 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser01/ | labuser01 | rgnet | labuser01 | 2013-05-06 21:42:15 | | 43 | rgnet | rsync://rgnet.rpki.net/rpki/rgnet/labuser03/ | labuser03 | rgnet | labuser03 | 2013-05-06 21:42:15 | +-----------+---------------+--------------------------------------------------+---------------+----------------+---------------+---------------------+ }}}

Trac comment by sra on 2013-05-06T21:44:16Z

[sraustein]

Doh, breaking and re-creating parent/child relationship blew away allocations from rgnet to labuser03, those need to be re-added in GUI or via the lab reset script. Will leave that for Randy.

Trac comment by sra on 2013-05-06T21:51:29Z

[sraustein]

rgnet delegated 98.128.3.0/24 to labuser03. labuser03 deleted the roa for the delegated prefix. no alerts.

Trac comment by randy on 2013-05-07T02:02:52Z

[sraustein]

Does that qualify as "fixed"?

Trac comment by sra on 2013-05-07T02:50:56Z

[sraustein]

seems to be

Trac comment by randy on 2013-05-07T03:15:27Z

[sraustein]

Closed with resolution fixed