search

job / draft-sidrops-rrdp-desynchronization

Detecting RRDP Session Desynchronization
1 stars 1 forks source link

Get data on how often desynchronization occurs #3

Open ties opened 7 months ago

ties commented 7 months ago

For the motivation of this document it would be good to know how often this de-synchronization occurs.

I analysed ~142k deltas but did not find duplicates in there. Unfortunately this negative result does not mean that no duplicate deltas were present in notification files: It just means that I did not find them, this could be due to the data gathering not downloading them, or a issue in my analysis.

job commented 7 months ago 
$ pwd
/var/www/htdocs/rpkidata/comparison.rpkiviews.org

$ find 2023 2024 -type f -name '*rpki-client.log.txt' | xargs fgrep 'unexpected delta'
2023/07/12/20230712T155201Z-rpki-client.log.txt:Jul 12 15:52:03 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 8596e6d7-daec-4db3-b83f-c8621b4a2536#1741 unexpected delta mutation (expected 25F6C73C58006ADA5FCB63E1B15D52B70610BB1792C06DDBF00B8ED2A2456434, got 462C2E3524B0712365831096A8F0A8B1BEE4A13C57FA58EA4B73F88BA456023C)
2023/08/22/20230822T200401Z-rpki-client.log.txt:Aug 22 20:04:13 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 77f03228-1baa-4fe2-b688-1b4166e22cff#758 unexpected delta mutation (expected EA8F2DD26D1BD52647CB917A640DC3683736FCA2A41A7CE8BA0DF7A1419A70AA, got B3041922B84821D82F102CD5CC376D258FA80E5206C08A112A77DF366993097F)
2023/09/06/20230906T000801Z-rpki-client.log.txt:Sep 06 00:08:04 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 77f03228-1baa-4fe2-b688-1b4166e22cff#2318 unexpected delta mutation (expected 6239EAC07FE10DCA9A422EE36E56C7558DE9E1B6CAF3593B7D7317D53AC6D466, got C5A4889CAEE8AE4135BA08CB7102D33A41D47AB5A02FA158ADD30553F8CFC07F)
2023/09/06/20230906T200801Z-rpki-client.log.txt:Sep 06 20:08:02 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 77f03228-1baa-4fe2-b688-1b4166e22cff#2409 unexpected delta mutation (expected BBBCBD768FDD28F87DC8500BF11D424F198FDA9C3C4136C241A82E591E168475, got 28F3A3A4E041BD1578EE51F8093187DFF78DB5BABC4F62FD0E256B8F4C977E2A)
2023/09/13/20230913T000601Z-rpki-client.log.txt:Sep 13 00:06:13 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 377fd663-52d9-4ae9-9b60-f966fbbda31a#164 unexpected delta mutation (expected F3CB0224E491E11FC60651C53FF6EA1CD7138E43383D2E0317DA7BF0A98C645A, got ABBDDEBD6CA81B6EDC8BA084A5340B75F1A2020052FCB1465EF01C2DE88204F0)
2023/09/25/20230925T001801Z-rpki-client.log.txt:Sep 25 00:18:10 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: e872dca0-2791-4522-82e6-9db076b260ec#302 unexpected delta mutation (expected 8BBF27BB0DF954435BA8F432D6AD31FC457D2B51B2219215E20429EDD2E0FA20, got EC23529170F67FF9D7BF7DB658D4EC7437E5941E7FB9FFF12CEBB0D7CB7055A5)
2024/01/04/20240104T014201Z-rpki-client.log.txt:Jan 04 01:45:44 rpki-client: https://cloudie.rpki.app/rrdp/notification.xml: d92cf37b-c955-4272-bbe6-efda084ad2de#2802 unexpected delta mutation (expected E1505DB0406D301F2BC4B17CDF12840C0F1EC52A1011309E37D137058AEB1FFC, got F1386D959C8EE9982FC850891A5103F8959810E489C483AAE6244E1DF8C1407A)
ties commented 7 months ago

From before my data collection period. Makes sense

On Thu, Feb 1, 2024, 11:02 Job Snijders @.***> wrote:

dango$ find 2023 2024 -type f -name '*rpki-client.log.txt' | xargs fgrep 'unexpected delta' | tee /tmp/l 2023/07/12/20230712T155201Z-rpki-client.log.txt:Jul 12 15:52:03 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 8596e6d7-daec-4db3-b83f-c8621b4a2536#1741 unexpected delta mutation (expected 25F6C73C58006ADA5FCB63E1B15D52B70610BB1792C06DDBF00B8ED2A2456434, got 462C2E3524B0712365831096A8F0A8B1BEE4A13C57FA58EA4B73F88BA456023C) 2023/08/22/20230822T200401Z-rpki-client.log.txt:Aug 22 20:04:13 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 77f03228-1baa-4fe2-b688-1b4166e22cff#758 unexpected delta mutation (expected EA8F2DD26D1BD52647CB917A640DC3683736FCA2A41A7CE8BA0DF7A1419A70AA, got B3041922B84821D82F102CD5CC376D258FA80E5206C08A112A77DF366993097F) 2023/09/06/20230906T000801Z-rpki-client.log.txt:Sep 06 00:08:04 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 77f03228-1baa-4fe2-b688-1b4166e22cff#2318 unexpected delta mutation (expected 6239EAC07FE10DCA9A422EE36E56C7558DE9E1B6CAF3593B7D7317D53AC6D466, got C5A4889CAEE8AE4135BA08CB7102D33A41D47AB5A02FA158ADD30553F8CFC07F) 2023/09/06/20230906T200801Z-rpki-client.log.txt:Sep 06 20:08:02 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 77f03228-1baa-4fe2-b688-1b4166e22cff#2409 unexpected delta mutation (expected BBBCBD768FDD28F87DC8500BF11D424F198FDA9C3C4136C241A82E591E168475, got 28F3A3A4E041BD1578EE51F8093187DFF78DB5BABC4F62FD0E256B8F4C977E2A) 2023/09/13/20230913T000601Z-rpki-client.log.txt:Sep 13 00:06:13 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: 377fd663-52d9-4ae9-9b60-f966fbbda31a#164 unexpected delta mutation (expected F3CB0224E491E11FC60651C53FF6EA1CD7138E43383D2E0317DA7BF0A98C645A, got ABBDDEBD6CA81B6EDC8BA084A5340B75F1A2020052FCB1465EF01C2DE88204F0) 2023/09/25/20230925T001801Z-rpki-client.log.txt:Sep 25 00:18:10 rpki-client: https://rrdp.lacnic.net/rrdp/notification.xml: e872dca0-2791-4522-82e6-9db076b260ec#302 unexpected delta mutation (expected 8BBF27BB0DF954435BA8F432D6AD31FC457D2B51B2219215E20429EDD2E0FA20, got EC23529170F67FF9D7BF7DB658D4EC7437E5941E7FB9FFF12CEBB0D7CB7055A5) 2024/01/04/20240104T014201Z-rpki-client.log.txt:Jan 04 01:45:44 rpki-client: https://cloudie.rpki.app/rrdp/notification.xml: d92cf37b-c955-4272-bbe6-efda084ad2de#2802 unexpected delta mutation (expected E1505DB0406D301F2BC4B17CDF12840C0F1EC52A1011309E37D137058AEB1FFC, got F1386D959C8EE9982FC850891A5103F8959810E489C483AAE6244E1DF8C1407A)

— Reply to this email directly, view it on GitHub https://github.com/job/draft-sidrops-rrdp-desynchronization/issues/3#issuecomment-1920951924, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABQTESOIPSIJKTMDHPYAI3YRNR4HAVCNFSM6AAAAABCUOWSF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRQHE2TCOJSGQ . You are receiving this because you authored the thread.Message ID: @.***>

job commented 7 months ago

Isn’t cloudie.rpki.app a krill instance? That one surprises me a bit to see in the list

ties commented 7 months ago

It looks like a krill instance from the repository structure. Looking at the log and the content of the RRDP files there was some time-travel there:

0E357DDA04B8107626124F52CFC2439B76D85AF9.mft

2024-01-04T0131Z #2806 not-modified https://comparison.rpkiviews.org/2024/01/04/20240104T013101Z-rpki-client.log.txt
2024-01-04T0142Z #2802 mutates https://comparison.rpkiviews.org/2024/01/04/20240104T014201Z-rpki-client.log.txt

Looking at the delta:

$ wget https://comparison.rpkiviews.org/rrdpdata/cloudie.rpki.app/rrdp/d92cf37b-c955-4272-bbe6-efda084ad2de/2802/9f3ada1aec3dea98/delta.xml.gz
$ gunzip delta.xml.gz
$ poetry run python -m rrdp_tools.cli reconstruct-repo ~/Downloads/delta.xml ~/Downloads/delta-2802/ --create-target
...
$ rpki-client -f 0E357DDA04B8107626124F52CFC2439B76D85AF9.mft
rpki-client: parse file ta/afrinic/AfriNIC.cer: No such file or directory
rpki-client: parse file ta/lacnic/rta-lacnic-rpki.cer: No such file or directory
rpki-client: parse file ta/apnic/apnic-rpki-root-iana-origin.cer: No such file or directory
rpki-client: parse file ta/ripe/ripe-ncc-ta.cer: No such file or directory
File:                     0E357DDA04B8107626124F52CFC2439B76D85AF9.mft
Hash identifier:          VrsuAh10bt67kdw4HHNeT0zAJJQmv+NiE/yyCxzX+Vk=
rpki-client: parse file cloudie-repo.rpki.app/repo/CLOUDIE-RPKI/0/0E357DDA04B8107626124F52CFC2439B76D85AF9.crl: No such file or directory
rpki-client: parse file rpki-rps.arin.net/repository/8a848adf850d063e0185755c91be3f9d/2/0E357DDA04B8107626124F52CFC2439B76D85AF9.cer: No such file or directory
rpki-client: failed to build authority chain
Subject key identifier:   73:B2:4A:22:7A:E3:B0:56:37:57:D8:F3:46:1D:48:BD:F7:9D:9D:F1
Authority key identifier: 0E:35:7D:DA:04:B8:10:76:26:12:4F:52:CF:C2:43:9B:76:D8:5A:F9
Certificate issuer:       /CN=0E357DDA04B8107626124F52CFC2439B76D85AF9
Certificate serial:       30CC31EB7A3FCC8AB29F56A912AC87DE474C56B5
Authority info access:    rsync://rpki-rps.arin.net/repository/8a848adf850d063e0185755c91be3f9d/2/0E357DDA04B8107626124F52CFC2439B76D85AF9.cer
Subject info access:      rsync://cloudie-repo.rpki.app/repo/CLOUDIE-RPKI/0/0E357DDA04B8107626124F52CFC2439B76D85AF9.mft
Manifest number:          0467
Signing time:             Wed 03 Jan 2024 09:48:02 +0100
Manifest this update:     Wed 03 Jan 2024 09:43:02 +0100
Manifest next update:     Thu 04 Jan 2024 13:01:02 +0100

If we look in the snapshot:

$ wget https://comparison.rpkiviews.org/rrdpdata/cloudie.rpki.app/rrdp/d92cf37b-c955-4272-bbe6-efda084ad2de/2802/036f98547dd26dcb/snapshot.xml.gz
$ gunzip snapshot.xml.gz
$ poetry run python -m rrdp_tools.cli reconstruct-repo ~/Downloads/snapshot.xml ~/Downloads/snapshot-2802/ --create-target
...
$ rpki-client -f 0E357DDA04B8107626124F52CFC2439B76D85AF9.mft
rpki-client: parse file ta/afrinic/AfriNIC.cer: No such file or directory
rpki-client: parse file ta/lacnic/rta-lacnic-rpki.cer: No such file or directory
rpki-client: parse file ta/apnic/apnic-rpki-root-iana-origin.cer: No such file or directory
rpki-client: parse file ta/ripe/ripe-ncc-ta.cer: No such file or directory
File:                     0E357DDA04B8107626124F52CFC2439B76D85AF9.mft
Hash identifier:          lql+Q2tkoduzfb+23A7KWVobBj9O4cSaWvqj6SKxnR8=
rpki-client: parse file cloudie-repo.rpki.app/repo/CLOUDIE-RPKI/0/0E357DDA04B8107626124F52CFC2439B76D85AF9.crl: No such file or directory
rpki-client: parse file rpki-rps.arin.net/repository/8a848adf850d063e0185755c91be3f9d/2/0E357DDA04B8107626124F52CFC2439B76D85AF9.cer: No such file or directory
rpki-client: failed to build authority chain
Subject key identifier:   AB:4B:BE:2E:F8:8A:2D:5E:3B:A0:AF:D0:7C:F2:F5:35:13:13:48:80
Authority key identifier: 0E:35:7D:DA:04:B8:10:76:26:12:4F:52:CF:C2:43:9B:76:D8:5A:F9
Certificate issuer:       /CN=0E357DDA04B8107626124F52CFC2439B76D85AF9
Certificate serial:       01BAF78F52CF01494FE51ED8F0C950E065716D46
Authority info access:    rsync://rpki-rps.arin.net/repository/8a848adf850d063e0185755c91be3f9d/2/0E357DDA04B8107626124F52CFC2439B76D85AF9.cer
Subject info access:      rsync://cloudie-repo.rpki.app/repo/CLOUDIE-RPKI/0/0E357DDA04B8107626124F52CFC2439B76D85AF9.mft
Manifest number:          0467
Signing time:             Thu 04 Jan 2024 01:36:54 +0100
Manifest this update:     Thu 04 Jan 2024 01:31:54 +0100
Manifest next update:     Fri 05 Jan 2024 05:08:54 +0100
job commented 7 months ago

@ties seems a backup was restored