Demo using Collaborative Launch feature at 1702994400 (Kartograf 0.4.1)

fjahr commented 10 months ago

In Kartograf 0.3.1 a new feature was added that allows users to start the mapping process at the exact same time at a previously agreed timestamp. This has shown some promise to result in the same or at least very similar results for the final mapping file across several participants. If multiple participants get the same result independently, trust in the ASMap file is further minimized as the input data generation is effectively federated. From the point of the creation of the input data, the process of arriving at the final ASMap file is reproducible with open-source tools and thus completely trustless.

As a demo of how this could work in the future, I propose the following timestamp to start a mapping process collaboratively: 1702994400. This is Tuesday, December 19, 2023 2:00:00 PM GMT, the same time of day at which the Bitcoin Core IRC meeting happens on Thursdays.

What you need to do to participate:

If you haven't used it already, get the latest version of Kartograf, install instruction are in the README. To ensure everything is set up correctly, maybe run ./run map once before setting the collaborative launch. If you have used Kartograf in the past you will probably still need to upgrade to the latest version, 0.3.1 at least.
When everything is set up you just need to run ./run map -w=1702994400 -irr -rv. The program runs and waits until the timestamp is hit, you just need to ensure that your computer is running until then and let it finish the process.
When the process is finished you will see a hash of the final result printed to the screen, please post it here as a comment.

A few notes, especially if you have tried Kartograf previously:

Runtime: Kartograf now uses multithreading, which means runtime can now be dramatically shorter than previously but it depends on the machine a lot and during certain periods you will see a lot of CPU usage. For reference, on a 2-year-old Macbook Pro with basic specs I expect to see ~3h for everything.
Network: Kartograf has been restructured to download everything it needs as fast as possible in the beginning since this also helps the chances of getting matching final results among participants. You should see a big spike in network activity in the beginning for a few minutes and then the tools runs basically fully offline. Since this is quite a bit of data and finishing fast means higher chance of success, I wouldn't recommend tethering over your mobile maybe. On the other hand, we also want to stress test this feature so if you only have a slow connection available, please still participate if you can!
Data: This is a full run collecting RPKI, IRR and Collectors data (from Routeviews). In previous demos didn't do this because of the long runtime.

Let me know if there are any questions. If you plan to participate, happy to see you confirm beforehand here as well.

brunoerg commented 10 months ago

I can join it.

Sjors commented 10 months ago

I'll join too!

Also I'm going to do a test run in about 40 minutes at 15:00 UTC (1702652400).

brunoerg commented 10 months ago

Also I'm going to do a test run in about 40 minutes at 15:00 UTC (1702652400).

I will do the same.

0xB10C commented 10 months ago

I'll join too.

brunoerg commented 10 months ago

@Sjors, 1702652400 is December 19, no?

fjahr commented 10 months ago

@Sjors, 1702652400 is December 19, no?

Seems to be today per https://www.epochconverter.com/

brunoerg commented 10 months ago

Sorry, my bad.. it's right.

Sjors commented 10 months ago

Only took 6 minutes it seems (I was away from the computer for a few hours, so didn't check):

./run map -w 1702652400

``` --- Start Kartograf --- Using rpki-client version 8.5. Coordinated launch mode: Waiting until 1702652400 (2023-12-15 16:00:00 CET) to launch mapping process. The epoch for this run is: 1702652400 (2023-12-15 15:00:00 UTC, local: 2023-12-15 16:00:00 CET) --- Fetching RPKI --- Downloading RPKI Data ...finished in 0:01:19.281949 --- Validating RPKI --- Validating RPKI ROAs 207653 raw RKPI DB entries validated and saved to /home/sjors/dev/kartograf/out/1702652400/rpki/rpki_raw.json ...finished in 0:04:13.281200 --- Parsing RPKI --- Parsing 207653 ROAs Output: 450394 Duplicates: 55207 Invalids: 20161 Incompletes: 0 ...finished in 0:00:47.687450 --- Sorting results --- ...finished in 0:00:02.180082 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 5c2f15b28c1d67aea2ea78bbc62a7aa0b1d9388f0ba8551690c0d82522ca2365 Total runtime: 0:06:22.438704 ```

brunoerg commented 10 months ago

I'm running with -rv -irr flags, so it will take longer.

Sjors commented 10 months ago

Oh oops, forgot those flags. I assume you missed my timestamp? Shall we try again in an hour, 18:00 UTC? 1702663200

brunoerg commented 10 months ago

Yes, we can try again 18:00 UTC. Will schedule here.

jurraca commented 10 months ago

I'll join for Tuesday 19th too.

Sjors commented 10 months ago

12ae155cf97db166a477acd10e9f67da6b6d926cf619ca7ac5528303ad19460e

./run map -w 1702663200 -irr -rv

``` ./run map -w 1702663200 -irr -rv --- Start Kartograf --- Using rpki-client version 8.5. Coordinated launch mode: Waiting until 1702663200 (2023-12-15 19:00:00 CET) to launch mapping process. The epoch for this run is: 1702663200 (2023-12-15 18:00:00 UTC, local: 2023-12-15 19:00:00 CET) --- Fetching RPKI --- Downloading RPKI Data ...finished in 0:01:17.939409 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:00:38.396658 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231213-1400.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231214-1200.pfx2as.gz ...finished in 0:00:06.225979 --- Validating RPKI --- Validating RPKI ROAs 207641 raw RKPI DB entries validated and saved to /home/sjors/dev/kartograf/out/1702663200/rpki/rpki_raw.json ...finished in 0:04:12.834675 --- Parsing RPKI --- Parsing 207641 ROAs Output: 450373 Duplicates: 55212 Invalids: 20163 Incompletes: 0 ...finished in 0:00:45.886287 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/sjors/dev/kartograf/out/1702663200/irr/arin.db Found valid entries: 102187 Parsing /home/sjors/dev/kartograf/out/1702663200/irr/afrinic.db Found valid entries: 112280 Parsing /home/sjors/dev/kartograf/out/1702663200/irr/lacnic.db Found valid entries: 12258 Parsing /home/sjors/dev/kartograf/out/1702663200/irr/ripe.db.route Found valid entries: 405438 Parsing /home/sjors/dev/kartograf/out/1702663200/irr/apnic.db.route6 Found valid entries: 328706 Parsing /home/sjors/dev/kartograf/out/1702663200/irr/ripe.db.route6 Found valid entries: 196950 Parsing /home/sjors/dev/kartograf/out/1702663200/irr/apnic.db.route Found valid entries: 692833 ...finished in 0:02:49.800866 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 115661 / 115661 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115661 / 115661 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115661 / 115661 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115661 / 115661 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115660 / 115660 | c Finished filtering! Originally 1850564 entries filtered down to 523466 Merging base file with filtered extra file ...finished in 1:14:11.423314 --- Parsing Routeviews pfx2as --- Unzipping /home/sjors/dev/kartograf/data/1702663200/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/sjors/dev/kartograf/out/1702663200/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/sjors/dev/kartograf/data/1702663200/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/sjors/dev/kartograf/out/1702663200/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/sjors/dev/kartograf/out/1702663200/collectors/pfx2asn.txt ...finished in 0:01:41.340878 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 74320 / 74320 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74319 / 74319 | Finished filtering! Originally 1189105 entries filtered down to 364028 Merging base file with filtered extra file ...finished in 1:48:57.141565 --- Sorting results --- ...finished in 0:00:07.112735 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 12ae155cf97db166a477acd10e9f67da6b6d926cf619ca7ac5528303ad19460e Total runtime: 3:14:49.481152 ```

Looking forward to the 19th!

brunoerg commented 10 months ago

Here is processing yet, I think I need a better computer :(

fjahr commented 10 months ago

Here is processing yet, I think I need a better computer :(

What are you using and how far did you get compared to Sjors logs? It would be helpful to know some benchmarks so we can give more precise estimates for runtime and system requirements in the Kartograf readme. Thanks!

brunoerg commented 10 months ago

What are you using and how far did you get compared to Sjors logs? It would be helpful to know some benchmarks so we can give more precise estimates for runtime and system requirements in the Kartograf readme. Thanks!

It's a macbook air M1 (8GB Ram) and It didn't finish yet, it's merging Routeviews and base data. As soon as it finishes I will post here the results.

brunoerg commented 10 months ago

My result:

--- Start Kartograf ---

Using rpki-client version 8.6.
Coordinated launch mode: Waiting until 1702663200 (2023-12-15 15:00:00 -03) to launch mapping process.
The epoch for this run is: 1702663200 (2023-12-15 18:00:00 UTC, local: 2023-12-15 15:00:00 -03)

--- Fetching RPKI ---

Downloading RPKI Data
...finished in 0:02:36.292543

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz
Downloading arin.db.gz
Downloading lacnic.db.gz
Downloading ripe.db.route.gz
Downloading ripe.db.route6.gz
...finished in 0:01:47.128586

--- Fetching Routeviews pfx2as ---

Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231213-1400.pfx2as.gz
Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231214-1200.pfx2as.gz
...finished in 0:00:09.234287

--- Validating RPKI ---

Validating RPKI ROAs
117591 raw RKPI DB entries validated and saved to /Users/brunogarcia/projects/kartograf/out/1702663200/rpki/rpki_raw.json
...finished in 0:49:51.927383

--- Parsing RPKI ---

Parsing 117591 ROAs
Output: 362949
Duplicates: 34085
Invalids: 21273
Incompletes: 0
...finished in 0:01:03.393675

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/lacnic.db
Found valid entries: 12258
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/ripe.db.route
Found valid entries: 405438
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/apnic.db.route6
Found valid entries: 328706
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/apnic.db.route
Found valid entries: 692833
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/arin.db
Found valid entries: 102187
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/afrinic.db
Found valid entries: 112280
Parsing /Users/brunogarcia/projects/kartograf/out/1702663200/irr/ripe.db.route6
Found valid entries: 196950
...finished in 0:04:50.545528

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   231321 /   231321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231321 /   231321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231321 /   231321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231321 /   231321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231320 /   231320 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231320 /   231320 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231320 /   231320 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   231320 /   231320 |

Finished filtering! Originally 1850564 entries filtered down to 559014
Merging base file with filtered extra file
...finished in 3:28:58.637742

--- Parsing Routeviews pfx2as ---

Unzipping /Users/brunogarcia/projects/kartograf/data/1702663200/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /Users/brunogarcia/projects/kartograf/out/1702663200/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /Users/brunogarcia/projects/kartograf/data/1702663200/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /Users/brunogarcia/projects/kartograf/out/1702663200/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /Users/brunogarcia/projects/kartograf/out/1702663200/collectors/pfx2asn.txt
...finished in 0:02:14.278093

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   148639 /   148639 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   148638 /   148638 |

Finished filtering! Originally 1189105 entries filtered down to 427670
Merging base file with filtered extra file
...finished in 6:02:07.496718

--- Sorting results ---

...finished in 0:00:07.938615

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 5628b36ad242e80361de490379bb10c53a176e33fbdab0c89eee18f5d270cb54
Total runtime: 10:33:51.267698

Sjors commented 10 months ago

Different hash :-(

I might be useful to also print out a hash of the downloaded material, so we can tell the difference between a non-deterministic download vs. some processing step.

fjahr commented 10 months ago

Different hash :-(

I might be useful to also print out a hash of the downloaded material, so we can tell the difference between a non-deterministic download vs. some processing step.

Yes, we can add something like this though in practice the issues were almost always with RPKI and I can see in the logs that the numbers are different there as well between both of your runs. One difference is that the rpki-client version is different, though I have tested 8.5 vs 8.6 explicitly in the past and arrived at the same hash. I will look at the code a bit more to see if I can improve it. At least I will try to make the logging for the RPKI part more detailed.

One question to both of you, @Sjors @brunoerg , were either of you using a VPN or something similar? With rpki-client we are downloading the ROA repositories from lots of different sources and I am curious if some of them maybe block some IPs to prevent DoS and that might lead to different results.

fjahr commented 10 months ago

@Sjors @brunoerg maybe each you could run the following from within your Kartograf directory and post the results:

$ find data/1702663200/rpki -mindepth 1 -maxdepth 1 -type d -exec sh -c 'printf "%s: " "{}"; find "{}" -type f | wc -l' \;

Example

``` $ find data/1702250850/rpki -mindepth 1 -maxdepth 1 -type d -exec sh -c 'printf "%s: " "{}"; find "{}" -type f | wc -l' \; data/1702250850/rpki/rpki-rsync.mnihyc.com: 3 data/1702250850/rpki/pub.krill.ausra.cloud: 8 data/1702250850/rpki/rpki.nap.re: 17 data/1702250850/rpki/rpki.apernet.io: 20 data/1702250850/rpki/rpki.sub.apnic.net: 124 data/1702250850/rpki/rpki-repository.nic.ad.jp: 5403 data/1702250850/rpki/rpki.akrn.net: 10 data/1702250850/rpki/rpki.admin.freerangecloud.com: 88 data/1702250850/rpki/rpkica.twnic.tw: 4610 data/1702250850/rpki/rov-measurements.nlnetlabs.net: 2 data/1702250850/rpki/rpki.cnnic.cn: 1105 data/1702250850/rpki/rpki.cc: 115 data/1702250850/rpki/rpki.apnic.net: 31245 data/1702250850/rpki/rpki.rand.apnic.net: 8 data/1702250850/rpki/.rrdp: 23452 data/1702250850/rpki/r.magellan.ipxo.com: 129 data/1702250850/rpki/rpki.qs.nu: 0 data/1702250850/rpki/krill.accuristechnologies.ca: 12 data/1702250850/rpki/rpki.komorebi.network: 15 data/1702250850/rpki/rpki.zappiehost.com: 38 data/1702250850/rpki/rsync.paas.rpki.ripe.net: 3414 data/1702250850/rpki/repo.kagl.me: 23 data/1702250850/rpki/rpki.folf.systems: 4 data/1702250850/rpki/ca.rg.net: 14 data/1702250850/rpki/rpki.pedjoeang.group: 14 data/1702250850/rpki/cloudie-repo.rpki.app: 386 data/1702250850/rpki/rpki.owl.net: 28 data/1702250850/rpki/rpki.roa.net: 102 data/1702250850/rpki/repo-rpki.idnic.net: 11356 data/1702250850/rpki/rpki-rsync.us-east-2.amazonaws.com: 1192 data/1702250850/rpki/0.sb: 230 data/1702250850/rpki/.rsync: 0 data/1702250850/rpki/chloe.sobornost.net: 26 data/1702250850/rpki/rpki-rps.arin.net: 682 data/1702250850/rpki/rsync.krill.cloud: 21 data/1702250850/rpki/krill.rayhaan.net: 11 data/1702250850/rpki/rpki.afrinic.net: 11414 data/1702250850/rpki/rpki-repository.haruue.net: 4 data/1702250850/rpki/dev.tw: 34 data/1702250850/rpki/rpki.multacom.com: 3 data/1702250850/rpki/rpki.xindi.eu: 15 data/1702250850/rpki/rsync.rp.ki: 122 data/1702250850/rpki/rpki-01.pdxnet.uk: 46 data/1702250850/rpki/rpki.services.vm.n1.i.bm-x0.w420.net: 3 data/1702250850/rpki/sakuya.nat.moe: 5 data/1702250850/rpki/rpki.co: 282 data/1702250850/rpki/rsync.rpki.tianhai.link: 17 data/1702250850/rpki/repository.lacnic.net: 17786 data/1702250850/rpki/rpki.luys.cloud: 2 data/1702250850/rpki/rpkica.mckay.com: 0 data/1702250850/rpki/ta: 4 data/1702250850/rpki/pub.rpki.win: 0 data/1702250850/rpki/rpki.ssmidge.xyz: 0 data/1702250850/rpki/rpki.ripe.net: 94119 data/1702250850/rpki/rpki1.rpki-test.sit.fraunhofer.de: 0 data/1702250850/rpki/rpki-repo.registro.br: 20409 ```

Explanation: You see a very different number between your logs where it says "XXX raw RKPI DB entries validated", this is actually the number of files downloaded and validated as part of the sync process of rpki-client. The number doesn't have to match exactly because there is overlap but it seems like @brunoerg got only half the files @Sjors did. So that seems a lot like @brunoerg was blocked from downloading some of the files.

brunoerg commented 10 months ago

At least I will try to make the logging for the RPKI part more detailed.

See https://github.com/fjahr/kartograf/pull/10

One question to both of you, @Sjors @brunoerg , were either of you using a VPN or something similar?

I'm not using a VPN.

See some logs from rpki-client:

Downloading RPKI Data
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/uMx4HMq3HaLS0ZoP0TbJGEBE2zA.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/V00kEnto5oHJEhRaMMayIbP4KlA.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/S4D0bEIIq3jyH3EKKWI1-QYyTis.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/STmJqI9ygR8i60Gk6wwSdOHx2pA.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/6n6vYSDTEzssFOqYEf97HcuEQhE.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/BpTOBmDPIzc01Obno4jqMUHuRbk.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36505B2/0569917622D711ED862FD6E0F1222468/z1Kz6_gz2w85Tz77x4mC_9aJbxA.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/0YiYIkQP2ghuQ_3L-e-Gb9Uepbw.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/vJWioqaGyUWVsW4DMK6ROPdodIg.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36FE1EA/0781C770806F11EEADC9ED584AD9E6FC/m2alk7NXauyLeoxnrfp6kgsUOag.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36EE1C1/050BE1D635B311ED9721E3F0F1222468/YmFCTuhQuS5FxpB3tvSkzniKeJM.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36886B3/5A411F9EB8EF11EDA5B09DDAF1222468/mxi9dO1YDbl9W9qdlXDKFpcMy8w.mft: no valid mft available
rpki-client: rpki.afrinic.net/repository/member_repository/F36432B6/0E94184AB75911EDA15F2C90F1222468/e9fVVRg6gfgLprfZNoeRJF7Y0z4.mft: no valid mft available
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/7F4A98EA6E0511E89C0D6E4BF8AEA228/8C8E0334474411EEBEB3AC7D4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F36C06CB/D8FF6538D4F311ECB3714BD3F1222468/BAD292FE050511EE9502F55D4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/7ECF1BA08DD211EE87A95A714AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/8CF246408DD111EE9587976E4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/3840A1221A3711EE938AB2854AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/CB2116DE8DD211EE8DA482724AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/25528B2A1A3711EE902D8E854AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/672390CC8DD111EE9E9B1F6E4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/2E5C22808DD211EE994C94704AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/2D4E05688DD011EEB2123F6A4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/A24143588DD011EE94E1B76B4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/B52AD8168DD111EEB145056F4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/072B6B5A8DD011EEAD82E4694AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/41C9EE881A3711EE93B6C1854AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/DB682A108DD111EE816D6C6F4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/025648328DD211EE9A1E11704AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/D1E6A4B48DCF11EE9F763B694AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/81D67B301A3311EE8D92D47A4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/742A216E1A1811EEBD8AD0274AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/7B2E40EA8DD011EEB4EB1B6B4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/590B8C508DD211EE8B05FE704AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/F29B40908DD211EEB5AAFD724AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/17C7F6B28DD111EE8C6BF56C4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/664F48608DD311EE99387A744AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/537EED4C8DD011EEAD0DAB6A4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/CC51A858919611EE96054180D25BE465.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/F07DC0828DD011EE8EDA986C4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/4089CB488DD111EEA45EA96D4AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/9E587424014D11EEA00401144AD9E6FC.roa: certificate has expired
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/A45742A88DD211EEB06407724AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/407C6E568DD311EEB9F5DA734AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/1B11BEDC8DD311EE80F370734AD9E6FC.roa: certificate is not yet valid
rpki-client: .rrdp/2A58FCECAC1BEAAECDB0232275D0A971B146A47C3E64FEF2FA46A24F5F6B1821/rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1B5854BF8AEA228/C935CF608DD011EEA9B6316C4AD9E6FC.roa: certificate is not yet valid
rpki-client: Error retrieving https://rrdp.rpki.co/rrdp/notification.xml (199.189.157.4): 204 No Content
rpki-client: https://rrdp.rpki.co/rrdp/notification.xml: load from network failed, fallback to rsync
rpki-client: rsync.rp.ki/repo/misakaio/0/93A2409684873795AF4D62FA9A8DC444FFFF145A.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/XAuPdiukfVBYvN8ApfpCEKTwNwatELGc3BPYhuAS1NU/0/6814B3F024DAEE755B7936B1C642C16DDA26A597.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/DWRkNyV6kLLcFax9mtQQX9LyT8iQuEZZYAhwDZZD6uPm/0/8F0427D606216B6EBB3E3F4EF473B346F5678EF2.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/7YupLApyE8p1jaBaMRVsTWqpoTgQJtc4USTWh6VemeKt/0/068D66B7337D24F9C661DF44896AE6F9199F7F12.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/GhWaNHQ2vn85c7jLoDTvgcMkiCx5bptBJB2wbRHE6z6Z/0/E7D8FAB52F7674C1DFA353660F987AD06D90CB72.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/J4B76xgG4Yme91gGrMLsrYMpPjZgFRbWQQJnqUgcXjRL/0/841CE1539C5B29E2CC6DD52791EF8C91725DF71E.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/ExkTHHoibuMaUZHXoT2hDEdsa47LEcCB5j8XSYUNEZvc/0/E2616F599452FCA56E99B35E225001CC7FA3AEE1.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/HbM3GcD9vbhXuXt6czJdin5YCv92asdECneSsrp1de2Y/0/C42FE5312BD8A91F2735F094C65D949782FBC27C.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/sQoVXD3vwPhHETfDm3R6Z3KpEFeVN8FWpnwapiAKCys/0/7453716841B3C3BFA0F638368F0F8373F22FF193.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/FFXkLkaKXHerXLpfUAso736rQEenqMpiv6QehFwmU1vW/0/209EFCD0E737A4897116B0C61EAC64613B3DD25C.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/YJnRBUoLVrgV6vKoL7miUpgq5oLKknLdYFy4hidU3E6/0/3306C7E72FEBE86DFD81FEBCE8EF03052858258D.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/2XEiRJNUeUwvecXb6MhzMDiMJUoYaer87j3qycjTcWsx/0/C3C02E9BA820D2E706528BC873C80443D02CE667.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/HB3JfCJY1CAJQAUs6iewCcTX3Sgsbs52wyrSq7RbJQmb/0/D94E7CBA4B0D83E8A3FC9011D30F767BC2E91A88.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/Gvh2oYGds1N6jhtMSrDbf5Z4ikTcmqoZL9M772ZYzr1R/0/999929A8883AD7E8D9D48AC57A08777F5F0757DE.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/J84si9JcSgS27zrfUTaLWzjuNSp2XhE8v5W2RQ7HmqzJ/0/CCB097ACA8EE51D21DFBE397BEF1D78345EF82D9.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/DSYKkKJHK2jBaQQMdVdW9rn3rvVx9fsN1i2smfeXLbEw/0/66EC78F47EAA40A6ABC0E1EE29D03B4D906904A3.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/B38JVmzAZPbtnHax1ECvCk5eYgBgkGS1qYgBXZdPVzUv/0/70392AC965030959311AF7FE238B273C8513237D.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/FcdDg2RpqCNF6N3vbD38jVQeYasit4WedbZRMJDoJbsq/0/36D0000793E703A1FE20CA3A1A3DCA18DBB32512.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/8gzz5TKKNpn3554Gog6UamkabYLrg8PApTW9ucKind8X/0/4C3458A01AC950A9DE1A8367C6F3E96BAF09B332.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/57aLNec1j5Kc5Mw7u4MBiYA4AKUyjJ1UCcfzJRwCGChq/0/D2D2D71D988525073935916EEBCA45B03AB78E14.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/71n4hyruudaEtzgLbheu8dLtwQyPpt4vzs7Ubib11UQC/0/AE7F53DA3301847BE9FCFB81FBBD1276432FCFBF.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/HUhNmCfpVUBACmjML8mw9GhU8sZNRWnzbBtBiSJDYdqZ/0/E0789CE58447880EA95EE455273D4A9A367316AE.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/4Bhazj8bFxvp7P6PcZgYFqm3bPxEryax3Q9RaDQNJ3P2/0/167B7DC84B098DDC2B9CE4E175A003955D57A017.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/8pJ3ozxZHgTtWtXMJYEFh95fF7tzp1SDGui2KaDMhfEy/0/9ED789D38EBBF63414DC8442A12A4EECA933C2D3.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/3KTipCQhHAKEUbDjCq3SPWqHPmoS8rXxdT7aR5gSHg6a/0/638993B01814A951CED1B1D993C6A07C5CCF0F85.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/BySDVgjB4YQhRaJzvyZv1JjbBhPBCngPfV1m4uv3W5m6/0/38ACEB0E95DA61DEA33E6DD5CAD7213525CCBD3F.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/DGbbvoNiDiYwbyFGEwxoTiUd74GFF7sbnqrsFaDi8WY9/0/66EA09809D63B05E9ED21D386D077B26F209BAF0.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/AAt3oabeiPvTAn6SgTwsQKTDrkL1bMh37vRPojX8Euo3/0/88EE17763F7A68907774C16D18127ECEFBF0D22F.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/6tzCM28VmLcWb2bAwF2h5ezzquFUiomV6jrRd8B4WYg3/0/7A66C1F92D197DDA862630D98506042FC4388765.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/23xpAP8pNyStTAATFkrMuJ8UgMsfEantMTStjQkAhGzB/0/CD9DEE4C93F29912C6A33DDA76B2248C8920919B.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/7JpkQPMXVWFE6wKj3rTx2dpgMAQ7HqgCrJMteAgYMueb/0/EEA4689702AE1DF5FCEF8390154D16A268774697.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/725DE8VuC1LEty5HDPBofeiERCTC9K38u8UP6Sbh5w9m/0/87FAEEAC57A774E51C582291B9B253EF7E0ACD80.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/J7DbE98dPZ9CVi2THC1oEUEsrMu5jF5MefnG19nWTzjr/0/9F447BDED8BDE868DC45F27E97C658F459051FB1.mft: CRL is not yet valid
rpki-client: rpki-repo.registro.br/repo/7PD2Pnw1ZrREnV3rNk8azZj1Xy6vfte1WLMJ9gBPNNug/0/DE117DF6BCF361FA73A81FDEAE5B6CBC6DD0640D.mft: no valid mft available
rpki-client: rpki-repo.registro.br/repo/3jvYAq8YM7riRtxmbzYYs5T6365iEg9AieED6dUA4GkZ/0/514A50B19852F6D708C89D381E43814F905B4853.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/Yo8QeTbuKZ259i5qSYjeeU8okKAKU6L8ZFu1G5WLT1H/0/030782B34602605D54BBF4F7F2F0FD538D237019.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/AoBbwqxX91Dexh1TTqzx9X9ZoFC4T5TL48b2qRSroDYW/0/A09087E55846274477AB0E4597FD754DB0509203.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/BFyiZhJjLBKzwSp4CTSi6Waai4KRDDiqE4AenVLceBTP/0/031FEFB409D35AA6A96C41F6CA93915FE8125453.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/5PMqxGXi1ZE4QJkbJT4t5cW3Zb3SRWsCNsGYyBZdkGXx/0/331BAC3DAA6C6F9BA61FC564124F9DDE11DE1735.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/FsMwpoWQDE9Q2Q4aTFGHqv6jdg2m7YFcD69NqK6e2fKs/0/F5CED17C45BDB6812EA27970F7981A63C8129A61.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/5PrjjzoAUney4CivY5xf3ff8oY7XeuJk5UaLbsb9TiGm/0/CE6ECE6859413800410D4C90D373B1EF989710AE.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/C1UfijZUrTzvC2AsesQSkVkF1zSvohG8Zq1MPqGByHTi/0/D4F66783B8162E6EEF49FBBF2B38D9A73DAD342D.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/6tiHo2CiAn1raQbQtgTzRjrY1BA3DQrp9koLhWDhTeWc/0/AD788192EB28093737DE304FE942917605211E2D.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/5LrejS5jBuavbF5rEWzo2RaMFua58XSiVueernNU8qgo/0/D2F36DD90FA86F0FFBAB74E26A9E1095716B1526.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/RZvY2SqjHmXdaekeTuVbpqQoYpvMDFrYWCPrcPgsjWD/0/D78AB6EBF2B6BF018794346C88788B0C3780E915.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/5XyS2CTU6pc2Lj5Ds2jqPideGHfqxdwpiaM3Q9Drt2Uz/0/B8E89DF044EBCBA78AC8564577BE3AB96DB0A7A6.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/FRRMRZp7zCXMUVUZSWNrs3qsPoP3b7XNyC9Rjqmjv5uz/0/5D18FA4A88E7A19A02E06CD348D6BDF263B88E3B.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/9SGb9oSxuzEc4ccPuDzUts37AUMombhtX51Pe25s98Fh/0/B4B19FDC97B4A5BBEF71A74C356CFE34DFCE3B8E.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/GrnHqfqkxJWMnb1eai4aqkrGs5kKSDijfapnJMu2XoLw/0/50967DD0D6ED253035A8473574C998814467525B.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/As5LLBDZTfx5NtVPgKoipgr7F7h3VnHQsKm3bQ8kaVeZ/0/A1F3B986BE342BBD942C2BD3B1365D87CE66DB04.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/Gp1UdqATaq3Fo9UTJb4Gs2CkH5szopYM4Lxahzo7dYmo/0/0E1716F26A112DCECF5028FF8275AC7C14903B6A.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/F58Vor2eSQ1f18YGaqhimNdjyYoHnQKyKv8Wjyn68F76/0/AF09E970726EC817A0DBDD43D7C859051FE47BB4.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/5gwFr1qg3vRDHhpAm4f275fgUoUanYQZixZowj7NfKhh/0/95BD1D7A4D8F79CB1C18D3503C43DB800C851B1D.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/EF72MJe18spBAbLNVDuc6B3BipCfNKntQNwuhZAwe88o/0/9EDC995FF6A556494A4F1D432E8E0EC17708DBD6.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/Dn78cT7wxzcB97Kdj8BnX7f4QVqvT6eUptHbFw9nworY/0/059B9CAFBD3B621B271B030203E1FAF8AE30B3A8.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/FEMGmkL6CpWnue3WgXpNgnvssfiWJ2Gnoa5CqTbLvShv/0/813614C43A5B26F0DFD7BA0A3578F1F68B3A0AC5.mft: CRL has expired
rpki-client: rpki-repo.registro.br/repo/DqFJvcWMStduX9q8LGyde5EdQCWxAuGfz22A8StqsN54/0/B3F05F894198E42754B64504D55F25A1B1869C43.mft: CRL has expired
rpki-client: rpki.cnnic.cn/rpki/A9162E3D0000/2075/6YpZWGZfNKJqPcr5TGOzOgcFmlM.mft: CRL has expired
rpki-client: rpki.cnnic.cn/rpki/A9162E3D0000/142/7z6j5ovlIQHmUZX6cIQJVc858dU.mft: CRL has expired
rpki-client: rpki.cnnic.cn/rpki/A9162E3D0000/369/w_lVr2j6-y_dR6ydZ5DlAsCMphM.mft: CRL has expired
rpki-client: .rsync/rpki.co/repo/AS945/13/AS7719.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: Error retrieving https://rpki.ssmidge.xyz/rrdp/notification.xml (155.248.239.44): 502 Bad Gateway
rpki-client: https://rpki.ssmidge.xyz/rrdp/notification.xml: load from network failed, fallback to rsync
rpki-client: https://krill.ca-bc-01.ssmidge.xyz/rrdp/notification.xml (23.156.104.131): connect: Connection refused
rpki-client: https://krill.ca-bc-01.ssmidge.xyz/rrdp/notification.xml: load from network failed, fallback to rsync
rpki-client: rpki.0i1.eu: nodename nor servname provided, or not known
rpki-client: https://rpki.0i1.eu/rrdp/notification.xml: load from network failed, fallback to rsync
rsync: getaddrinfo: rpki.0i1.eu 873: nodename nor servname provided, or not known
rsync error: error in socket IO (code 10) at clientserver.c(139) [Receiver=3.2.7]
rpki-client: rsync rsync://rpki.0i1.eu/repo failed
rpki-client: .rsync/rpki.0i1.eu/repo: load from network failed, fallback to cache
rpki-client: https://rpkica.mckay.com/rrdp/notify.xml (51.75.161.87): TLS handshake: certificate verification failed: certificate has expired
rpki-client: https://rpkica.mckay.com/rrdp/notify.xml: load from network failed, fallback to rsync
rpki-client: rpki.caramelfox.net: nodename nor servname provided, or not known
rpki-client: https://rpki.caramelfox.net/rrpdp/notification.xml: load from network failed, fallback to rsync
rsync: getaddrinfo: rpki.caramelfox.net 873: nodename nor servname provided, or not known
rsync error: error in socket IO (code 10) at clientserver.c(139) [Receiver=3.2.7]
rpki-client: rsync rsync://rpki.caramelfox.net/repo failed
rpki-client: .rsync/rpki.caramelfox.net/repo: load from network failed, fallback to cache
rpki-client: https://pub.rpki.win/rrdp/notification.xml (23.150.40.69): connect timeout
rpki-client: https://pub.rpki.win/rrdp/notification.xml: load from network failed, fallback to rsync
rpki-client: rpki.co/repo/Mario/0/92591D776FA24CA7C6E2BAA2D0E5B3E80FEA898D.mft: no valid mft available
rsync error: timeout waiting for daemon connection (code 35) at socket.c(278) [Receiver=3.2.7]
rpki-client: rsync rsync://rpki.ssmidge.xyz/repo failed
rpki-client: .rsync/rpki.ssmidge.xyz/repo: load from network failed, fallback to cache
rsync error: timeout waiting for daemon connection (code 35) at socket.c(278) [Receiver=3.2.7]
rpki-client: rsync rsync://krill.ca-bc-01.ssmidge.xyz/repo failed
rpki-client: .rsync/krill.ca-bc-01.ssmidge.xyz/repo: load from network failed, fallback to cache
rpki-client: rsync.paas.rpki.ripe.net/repository/30df2d1b-1498-4686-91bb-64b582010328/0/D6424E531A5AD6AC80AF20F0D6DC2CB187322EC1.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/646b5ebc-f4ee-4c8e-80a2-17b3e3acf734/2/4AFA0771812E6637D8B18A1FF50F41F3498669A9.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/1231fd86-c539-46c7-89e9-f3756f3075fa/1/F8A59FD412EBD333DDB170E8FC2803D23252071D.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/b64c075f-eb10-4426-bb40-3a833fe0f9fb/1/7DDB99F73E7B582C352890E3AD5785D3679F8891.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/32bfd357-d83b-400a-8c46-4fbd1119f4a3/2/C5557C97D7BBFF47CA22AD5FB9F0589FAAF159BD.mft: no valid mft available
rpki-client: rsync.paas.rpki.ripe.net/repository/646b5ebc-f4ee-4c8e-80a2-17b3e3acf734/1/25297F0C4B0430F30A7878D51FB3B7AC11250092.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/a4c6bdc5-eb4e-4b6e-95f6-62790e57f3be/0/9DA23CC2CF88AE4585CF8AAF9A714A9AD2E6F198.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/ad0eb3cf-9a1b-4112-a607-ae98c2ab12a7/0/52F4CAA87D081FC25DDF117BE8A9FE990AA120BC.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/32bfd357-d83b-400a-8c46-4fbd1119f4a3/1/AFE88BB6B8C27B032AEF245A4A94479E52E7408D.mft: no valid mft available
rpki-client: rsync.paas.rpki.ripe.net/repository/240ce547-40ef-4f04-95be-088f3bc02503/0/10DAC6AAF940C7C5FF1B83ECA3526D46BA475093.mft: no valid mft available
rpki-client: rsync.paas.rpki.ripe.net/repository/c5592a54-4035-4970-86dc-3d1803b7b60f/0/945031CB22A35CDB6CE579CB70270EA7B3A89A27.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/23d5f682-b51b-4812-b8b1-430e38683748/0/BB0F8437DE6BB251821A7A5FB8111376F1709023.mft: CRL has expired
rpki-client: rpki.0i1.eu/repo/h45/0/7C5CDC3FD51653B5120B4EBEDD2FA382240A5868.mft: no valid mft available
rpki-client: rsync.paas.rpki.ripe.net/repository/d2288b0e-a1d0-4c1a-8ced-a057a6076a7f/0/AB4C1697B18C3C715553206C4C39BBE9E52F658B.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/5be80edf-4126-4dc2-b337-5db6193122d4/0/FDE2FA815558D20831FD46062AF9B78ABC8E302E.mft: CRL has expired
rpki-client: krill.stonham.info/repo/Stonham/2/AAD12F5CF3A816DC2055C6F3AF5E2E9204550B05.mft: CRL has expired
rsync error: timeout waiting for daemon connection (code 35) at socket.c(278) [Receiver=3.2.7]
rpki-client: rsync rsync://pub.rpki.win/repo failed
rpki-client: .rsync/pub.rpki.win/repo: load from network failed, fallback to cache
rpki-client: rpki.caramelfox.net/repo/CaramelFoxNetworks/0/3F605BF3BC22BF10A45CA2660CEF9AA7E29A5EC8.mft: no valid mft available
rpki-client: rsync.paas.rpki.ripe.net/repository/1231fd86-c539-46c7-89e9-f3756f3075fa/0/0D2FD5128ACF54403635A1A1F6AEEF373414BA3D.mft: CRL has expired
rpki-client: rpkica.mckay.com/rpki/MCnet/Jp4Tjp_GB5I1RfeaOGhKZNlDmAQ.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/ad0eb3cf-9a1b-4112-a607-ae98c2ab12a7/3/FC5EF1F664F6BB0E1AF7BDA53D67E0589B895E1F.mft: CRL has expired
rpki-client: .rsync/rpki.co/repo/moeqing/0/AS210979.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/moeqing/0/AS200105.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: rpki.ssmidge.xyz/repo/as199177/1/3002D8AC926433CA570E46BF86B5A45E61F5CCA0.mft: no valid mft available
rpki-client: rpki1.rpki-test.sit.fraunhofer.de/repo/rpki1_fraunhofer/0/DF5A31E54915EB10EB015ED8F0476D8C3F64FFF1.mft: CRL has expired
rpki-client: .rsync/rpki.co/repo/Mario/1/AS50580.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: krill.ca-bc-01.ssmidge.xyz/repo/SsmidgeLLC/0/5336A1DC9C53858F5D437551051BB214BF1C5ABB.mft: no valid mft available
rpki-client: .rsync/rpki.co/repo/Riffi/0/AS216412.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Riffi/0/AS199514.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Riffi/0/AS203868.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Riffi/0/AS200579.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS202400/2/AS59730.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS202400/2/AS202400.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS207466/0/AS207466.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS202400/3/AS202996.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Matt/0/AS64457.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Matt/0/AS202559.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Matt/3/AS49592.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/Misaka/0/AS203979.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/John/0/AS200879.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rrdp/7FCA1FB9413F98018210E7533C4C3C6B231599C6B44A6F85730392A79818A2EA/rsync.paas.rpki.ripe.net/repository/aa004ba1-419b-4db5-bbd3-5cca633cae3f/2/AS51019.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS202939/0/AS202939.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: pub.rpki.win/repo/rpkiwin/0/5FDA2DF4754418696CDBE1CA590C7D741463F56C.mft: no valid mft available
rpki-client: .rsync/rpki.co/repo/AS211380/0/AS211380.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: rpki.qs.nu/repo/cyntest/2/FF0595CB477C33B6B94C7E477213C842A6067D6B.mft: CRL has expired
rpki-client: rpki.ssmidge.xyz/repo/ssmidgellc/2/EFC33765381A3F6F5CD6EBEE3A113B0747F70D75.mft: no valid mft available
rpki-client: .rsync/rpki.co/repo/AS945/0/AS51930.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS945/6/AS57481.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS945/4/AS49867.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rrdp/436FC6BD7B32853E42FCE5FD95B31D5E3EC1C32C46B7518C2067D568E7EAC119/chloe.sobornost.net/rpki/RIPE-nljobsnijders/cb/5EjPZ8Kw2_h5hRqKpwmjdnq7Tq8.roa: AKI doesn't match Manifest AKI
rpki-client: .rrdp/436FC6BD7B32853E42FCE5FD95B31D5E3EC1C32C46B7518C2067D568E7EAC119/chloe.sobornost.net/rpki/RIPE-nljobsnijders/cb/5EjPZ8Kw2_h5hRqKpwmjdnq7Tq8.roa: unable to get local issuer certificate
rpki-client: .rsync/rpki.co/repo/AS945/5/AS51818.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS945/5/AS59486.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS945/1/AS50058.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: .rsync/rpki.co/repo/AS945/1/AS47449.asa: ASPA: failed to parse ASProviderAttestation
rpki-client: rsync.paas.rpki.ripe.net/repository/bc90f4f6-f2c4-4882-91cf-9b8f0b38d7da/1/CEE50F8654426B5A60EE8C52A9AE958F8D77B848.mft: no valid mft available
rpki-client: rsync.paas.rpki.ripe.net/repository/1231fd86-c539-46c7-89e9-f3756f3075fa/2/ED7E642F8C5FB6C6E897FE906D788ADA542E8EED.mft: CRL has expired
rpki-client: krill.ca-bc-01.ssmidge.xyz/repo/SsmidgeLLC/1/7BBD0E669176F6F2E8BB8FC3104A8D23435175AE.mft: no valid mft available
rpki-client: krill.stonham.info/repo/Stonham/1/635C29FF238CC286AC1625A68EFCC04E2E460171.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/1231fd86-c539-46c7-89e9-f3756f3075fa/3/6085227D041EE38A5046749BFCC9B1327B980374.mft: CRL has expired
rpki-client: rsync.paas.rpki.ripe.net/repository/59183bf3-1acf-4a20-8fa4-b467c2c2260b/2/A5B9564158D3F545D38A5134082DABBC4CE4BBC0.mft: no valid mft available
rpki-client: krill.ca-bc-01.ssmidge.xyz/repo/AS199177/0/4173C015E8E1FED254D4938B7E69CB256CCF6936.mft: no valid mft available
rpki-client: rpki-repository.haruue.net/repo/YC3254-RPKI/2/3F0AC25D352C83DA8307594B98ED061BE8489682.mft: CRL has expired
...finished in 0:02:36.292543

Obs.: I'm in Brazil.

brunoerg commented 10 months ago

@Sjors @brunoerg maybe each you could run the following from within your Kartograf directory and post the results:

data/1702663200/rpki/rpki-rsync.mnihyc.com:        3
data/1702663200/rpki/pub.krill.ausra.cloud:        8
data/1702663200/rpki/rpki.nap.re:       17
data/1702663200/rpki/rpki.apernet.io:       20
data/1702663200/rpki/rpki.sub.apnic.net:      124
data/1702663200/rpki/rpki-repository.nic.ad.jp:     5422
data/1702663200/rpki/rpki.akrn.net:       10
data/1702663200/rpki/rpki.admin.freerangecloud.com:       88
data/1702663200/rpki/rpkica.twnic.tw:     4613
data/1702663200/rpki/rov-measurements.nlnetlabs.net:        2
data/1702663200/rpki/rpki.cnnic.cn:     1108
data/1702663200/rpki/rpki.cc:      115
data/1702663200/rpki/rpki.apnic.net:    31268
data/1702663200/rpki/rpki.rand.apnic.net:        8
data/1702663200/rpki/.rrdp:    23984
data/1702663200/rpki/r.magellan.ipxo.com:      125
data/1702663200/rpki/rpki.qs.nu:        0
data/1702663200/rpki/krill.accuristechnologies.ca:       12
data/1702663200/rpki/rpki.komorebi.network:       21
data/1702663200/rpki/rpki.zappiehost.com:       38
data/1702663200/rpki/rsync.paas.rpki.ripe.net:     3405
data/1702663200/rpki/repo.kagl.me:       23
data/1702663200/rpki/rpki.folf.systems:        4
data/1702663200/rpki/ca.rg.net:       14
data/1702663200/rpki/rpki.pedjoeang.group:       14
data/1702663200/rpki/cloudie-repo.rpki.app:      388
data/1702663200/rpki/rpki.owl.net:       28
data/1702663200/rpki/rpki.roa.net:       81
data/1702663200/rpki/repo-rpki.idnic.net:    11411
data/1702663200/rpki/rpki-rsync.us-east-2.amazonaws.com:     1192
data/1702663200/rpki/0.sb:      151
data/1702663200/rpki/.rsync:        0
data/1702663200/rpki/chloe.sobornost.net:       26
data/1702663200/rpki/rpki-rps.arin.net:      681
data/1702663200/rpki/rsync.krill.cloud:       17
data/1702663200/rpki/krill.rayhaan.net:       11
data/1702663200/rpki/rpki.afrinic.net:    11100
data/1702663200/rpki/rpki-repository.haruue.net:        0
data/1702663200/rpki/dev.tw:       34
data/1702663200/rpki/rpki.multacom.com:        3
data/1702663200/rpki/rpki.xindi.eu:       15
data/1702663200/rpki/rsync.rp.ki:      117
data/1702663200/rpki/rpki-01.pdxnet.uk:       46
data/1702663200/rpki/rpki.services.vm.n1.i.bm-x0.w420.net:        3
data/1702663200/rpki/sakuya.nat.moe:        5
data/1702663200/rpki/rpki.co:      282
data/1702663200/rpki/rsync.rpki.tianhai.link:       17
data/1702663200/rpki/repository.lacnic.net:    17863
data/1702663200/rpki/rpki.luys.cloud:        2
data/1702663200/rpki/rpkica.mckay.com:        0
data/1702663200/rpki/ta:        4
data/1702663200/rpki/pub.rpki.win:        0
data/1702663200/rpki/rpki.ssmidge.xyz:        0
data/1702663200/rpki/rpki.ripe.net:    93828
data/1702663200/rpki/rpki1.rpki-test.sit.fraunhofer.de:        0
data/1702663200/rpki/rpki-repo.registro.br:    20440

fjahr commented 10 months ago

@brunoerg Thanks, I don't really see anything worrying in the rpki-client logs. They are always extremely noisy but that doesn't mean something bad is happening.

FWIW, the numbers look very similar to the numbers from my example, now I'm curious how it will look for @Sjors.

Sjors commented 10 months ago

@fjahr I was also not using a VPN

My output...

``` data/1702663200/rpki/repo.kagl.me: 32 data/1702663200/rpki/rpki.qs.nu: 10 data/1702663200/rpki/rpki.services.vm.n1.i.bm-x0.w420.net: 3 data/1702663200/rpki/rpki.rand.apnic.net: 8 data/1702663200/rpki/rpki.komorebi.network: 21 data/1702663200/rpki/rsync.krill.cloud: 17 data/1702663200/rpki/rpki.apnic.net: 31268 data/1702663200/rpki/cloudie-repo.rpki.app: 460 data/1702663200/rpki/.rsync: 0 data/1702663200/rpki/repo-rpki.idnic.net: 11411 data/1702663200/rpki/rpki.0i1.eu: 0 data/1702663200/rpki/rpki-repository.nic.ad.jp: 5422 data/1702663200/rpki/rpki.sub.apnic.net: 208 data/1702663200/rpki/ca.rg.net: 14 data/1702663200/rpki/rpki.apernet.io: 20 data/1702663200/rpki/krill.ca-bc-01.ssmidge.xyz: 0 data/1702663200/rpki/rpki.ripe.net: 93828 data/1702663200/rpki/rpki.cc: 121 data/1702663200/rpki/rsync.rpki.tianhai.link: 17 data/1702663200/rpki/rsync.roa.tohunet.com:3873: 0 data/1702663200/rpki/rpki-repository.haruue.net: 0 data/1702663200/rpki/rpki-rps.arin.net: 1517 data/1702663200/rpki/rpki.afrinic.net: 11100 data/1702663200/rpki/rsync.rpki.co: 0 data/1702663200/rpki/rsync.rp.ki: 140 data/1702663200/rpki/rpki.roa.net: 146 data/1702663200/rpki/rov-measurements.nlnetlabs.net: 2 data/1702663200/rpki/ca.nat.moe: 97 data/1702663200/rpki/pub.rpki.win: 0 data/1702663200/rpki/rpki.ssmidge.xyz: 0 data/1702663200/rpki/rpki.luys.cloud: 2 data/1702663200/rpki/rpki.owl.net: 28 data/1702663200/rpki/rpki.arin.net: 96686 data/1702663200/rpki/rpki.xindi.eu: 15 data/1702663200/rpki/0.sb: 151 data/1702663200/rpki/rpki.cnnic.cn: 1108 data/1702663200/rpki/krill.rayhaan.net: 11 data/1702663200/rpki/rpki-rsync.mnihyc.com: 3 data/1702663200/rpki/rpki.tools.westconnect.ca: 0 data/1702663200/rpki/chloe.sobornost.net: 26 data/1702663200/rpki/.rrdp: 22672 data/1702663200/rpki/krill.accuristechnologies.ca: 58 data/1702663200/rpki/rpki-repo.registro.br: 20440 data/1702663200/rpki/rpki-01.pdxnet.uk: 58 data/1702663200/rpki/sakuya.nat.moe: 5 data/1702663200/rpki/rpki.zappiehost.com: 38 data/1702663200/rpki/rpki.co: 336 data/1702663200/rpki/ta: 5 data/1702663200/rpki/rpki.nap.re: 17 data/1702663200/rpki/dev.tw: 82 data/1702663200/rpki/rpki-rsync.us-east-2.amazonaws.com: 7792 data/1702663200/rpki/rpki.admin.freerangecloud.com: 169 data/1702663200/rpki/rpki.pedjoeang.group: 18 data/1702663200/rpki/rsync.paas.rpki.ripe.net: 3446 data/1702663200/rpki/repository.lacnic.net: 17863 data/1702663200/rpki/rpki1.rpki-test.sit.fraunhofer.de: 0 data/1702663200/rpki/krill.stonham.info: 0 data/1702663200/rpki/rpki.multacom.com: 15 data/1702663200/rpki/rpki.caramelfox.net: 3 data/1702663200/rpki/pub.krill.ausra.cloud: 8 data/1702663200/rpki/rpki.akrn.net: 14 data/1702663200/rpki/rpkica.twnic.tw: 4613 data/1702663200/rpki/r.magellan.ipxo.com: 125 data/1702663200/rpki/rpki.folf.systems: 4 ```

fjahr commented 10 months ago

Ok, @Sjors has much more ARIN data, both @brunoerg and me are missing the trust anchor as well. That is very weird, I will need to investigate why some of us are getting that data and others not...

fjahr commented 10 months ago

Sigh, ok, like every time I thought I was done... This is a bit of a political issue I guess. ARIN requires users to agree to their RPAs when using their TAL. That's why rpki-client ships with the other four TALs but not ARINs. They don't want to require users to opt-in to anything, see also https://github.com/rpki-client/rpki-client-portable/issues/80. Routinator, the other big RPKI validator, solves this by asking the user to pass a flag once during setup, see "Running Routinator" at https://labs.ripe.net/author/tashi_phuntsho_3/how-to-install-an-rpki-validator/.

I have fixed this now in Kartograf 0.3.1: we are now downloading all TALs ourselves and also safe them in the data dir, this also gives us additional security for reproducibility, though I don't think this would become a big issue. I am not so sure if we need something like Routinator does, for now, I have added a small note in the README.

So everyone please update to 0.3.1 before running on Tuesday. Thank you and sorry for the hassle! And special thanks to @Sjors and @brunoerg for finding bugs before the actual test even started :)

Lastly, I am not sure how @Sjors ended up having the ARIN TAL. My best guess is you previously ran an older version of rpki-client which still had the ARIN TAL included and then it was locally cached and used for verification.

brunoerg commented 10 months ago

Nice, @fjahr. I'll do another test run at 22h UTC (1702764000).

fjahr commented 10 months ago

Nice, @fjahr. I'll do another test run at 22h UTC (1702764000).

I'll join you for this one!

fjahr commented 10 months ago

@brunoerg I got 86269ff79b8842fd520d5491b362ab3425f714054e8ee1cb184b60944002fabd.

Detailed logs

``` $ ./run map -w=1702764000 -rv -irr --- Start Kartograf --- Using rpki-client version 8.5. Coordinated launch mode: Waiting until 1702764000 (2023-12-16 23:00:00 CET) to launch mapping process. The epoch for this run is: 1702764000 (2023-12-16 22:00:00 UTC, local: 2023-12-16 23:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1702764000/rpki/tals/afrinic.tal Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1702764000/rpki/tals/apnic.tal Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1702764000/rpki/tals/arin.tal Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1702764000/rpki/tals/lacnic.tal Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1702764000/rpki/tals/ripe.tal Downloading RPKI Data ...finished in 0:07:43.941778 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:01:07.916042 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231215-0400.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231215-1200.pfx2as.gz ...finished in 0:00:08.307598 --- Validating RPKI --- Validating RPKI ROAs 207584 raw RKPI ROA files found. 207584 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1702764000/rpki/rpki_raw.json ...finished in 0:36:00.984985 --- Parsing RPKI --- Parsing 207584 ROAs Result entries written: 362650 Duplicates found: 34065 Invalids found: 111733 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:40.727231 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/lacnic.db Found valid entries: 12259 Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/ripe.db.route Found valid entries: 405420 Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/apnic.db.route6 Found valid entries: 328713 Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/apnic.db.route Found valid entries: 692898 Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/arin.db Found valid entries: 102203 Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/afrinic.db Found valid entries: 112289 Parsing /Users/FJ/projects/python/kartograf/out/1702764000/irr/ripe.db.route6 Found valid entries: 196996 ...finished in 0:03:15.040537 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185069 / 185069 | Finished filtering! Originally 1850690 entries filtered down to 559325 Merging base file with filtered extra file ...finished in 0:54:09.221866 --- Parsing Routeviews pfx2as --- Unzipping /Users/FJ/projects/python/kartograf/data/1702764000/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1702764000/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/FJ/projects/python/kartograf/data/1702764000/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1702764000/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/FJ/projects/python/kartograf/out/1702764000/collectors/pfx2asn.txt ...finished in 0:01:56.510488 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 118908 / 118908 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118908 / 118908 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118907 / 118907 | Finished filtering! Originally 1189072 entries filtered down to 427722 Merging base file with filtered extra file ...finished in 1:41:41.490483 --- Sorting results --- ...finished in 0:00:07.776084 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 86269ff79b8842fd520d5491b362ab3425f714054e8ee1cb184b60944002fabd Total runtime: 3:26:53.717626 ```

brunoerg commented 10 months ago

@fjahr My result:

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 86269ff79b8842fd520d5491b362ab3425f714054e8ee1cb184b60944002fabd
Total runtime: 22:20:08.252168

Same hash 🚀

Sjors commented 10 months ago

I'll update and join tomorrow!

fjahr commented 10 months ago

A quick reminder that we're starting in about 2.5 hours. Please don't forget to update Kartofgraf to 0.3.1 first.

Then simply run:

./run map -w=1702994400 -irr -rv

fjahr commented 10 months ago

Actually 1.5 hours, time zones amirite... :)

0xB10C commented 10 months ago

Thanks for the reminder! I wasn't sure which version I'm on, but it seems like I'm up-to-date with 0.3.1 now. Might make sense to print the version during startup.

e.g.

--- Start Kartograf v0.3.1 ---

fjahr commented 10 months ago

Thanks for the reminder! I wasn't sure which version I'm on, but it seems like I'm up-to-date with 0.3.1 now. Might make sense to print the version during startup.

e.g.
--- Start Kartograf v0.3.1 ---

Yepp, makes sense since I we are already doing the same with rpki-client

Sjors commented 10 months ago

Hash: fac1322ab662097f3d9528b8a885a77589850fd1d73097007198a89edbce436a

Took about 3 hours, but the computer was doing some other heavy work.

Output for 1702994400...

``` ./run map -w=1702994400 -irr -rv --- Start Kartograf --- Using rpki-client version 8.5. Coordinated launch mode: Waiting until 1702994400 (2023-12-19 15:00:00 CET) to launch mapping process. The epoch for this run is: 1702994400 (2023-12-19 14:00:00 UTC, local: 2023-12-19 15:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/sjors/dev/kartograf/data/1702994400/rpki/tals/afrinic.tal Downloaded TAL for APNIC to /home/sjors/dev/kartograf/data/1702994400/rpki/tals/apnic.tal Downloaded TAL for ARIN to /home/sjors/dev/kartograf/data/1702994400/rpki/tals/arin.tal Downloaded TAL for LACNIC to /home/sjors/dev/kartograf/data/1702994400/rpki/tals/lacnic.tal Downloaded TAL for RIPE to /home/sjors/dev/kartograf/data/1702994400/rpki/tals/ripe.tal Downloading RPKI Data ...finished in 0:01:39.126418 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:00:32.377150 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231217-1200.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231218-1200.pfx2as.gz ...finished in 0:00:05.614021 --- Validating RPKI --- Validating RPKI ROAs 208233 raw RKPI ROA files found. 208233 RKPI ROAs validated and saved to /home/sjors/dev/kartograf/out/1702994400/rpki/rpki_raw.json ...finished in 0:04:22.497073 --- Parsing RPKI --- Parsing 208233 ROAs Result entries written: 450580 Duplicates found: 55311 Invalids found: 20503 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:46.910039 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/sjors/dev/kartograf/out/1702994400/irr/arin.db Found valid entries: 102228 Parsing /home/sjors/dev/kartograf/out/1702994400/irr/afrinic.db Found valid entries: 112301 Parsing /home/sjors/dev/kartograf/out/1702994400/irr/lacnic.db Found valid entries: 12216 Parsing /home/sjors/dev/kartograf/out/1702994400/irr/ripe.db.route Found valid entries: 405430 Parsing /home/sjors/dev/kartograf/out/1702994400/irr/apnic.db.route6 Found valid entries: 328726 Parsing /home/sjors/dev/kartograf/out/1702994400/irr/ripe.db.route6 Found valid entries: 197056 Parsing /home/sjors/dev/kartograf/out/1702994400/irr/apnic.db.route Found valid entries: 692914 ...finished in 0:02:53.941770 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115674 / 115674 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 115673 / 115673 | Finished filtering! Originally 1850783 entries filtered down to 523388 Merging base file with filtered extra file ...finished in 1:14:09.207811 --- Parsing Routeviews pfx2as --- Unzipping /home/sjors/dev/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/sjors/dev/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/sjors/dev/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/sjors/dev/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/sjors/dev/kartograf/out/1702994400/collectors/pfx2asn.txt ...finished in 0:01:46.065133 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74299 / 74299 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74298 / 74298 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 74298 / 74298 | Finished filtering! Originally 1188782 entries filtered down to 363741 Merging base file with filtered extra file ...finished in 1:48:30.838662 --- Sorting results --- ...finished in 0:00:07.045490 --- Finishing Kartograf --- The SHA-256 hash of the result file is: fac1322ab662097f3d9528b8a885a77589850fd1d73097007198a89edbce436a Total runtime: 3:14:55.034814 ```

fjahr commented 10 months ago

Unfortunately, I have a different hash :-/ 0df6aa452452d964f10ae2100e094f96e142051ff817fb75fabb90cdbdfd2bbf

Detailed logs

``` $ ./run map -w=1702994400 -irr -rv --- Start Kartograf --- Using rpki-client version 8.5. Coordinated launch mode: Waiting until 1702994400 (2023-12-19 15:00:00 CET) to launch mapping process. The epoch for this run is: 1702994400 (2023-12-19 14:00:00 UTC, local: 2023-12-19 15:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1702994400/rpki/tals/afrinic.tal Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1702994400/rpki/tals/apnic.tal Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1702994400/rpki/tals/arin.tal Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1702994400/rpki/tals/lacnic.tal Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1702994400/rpki/tals/ripe.tal Downloading RPKI Data ...finished in 0:08:01.736058 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:00:57.396436 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231217-1200.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231218-1200.pfx2as.gz ...finished in 0:00:06.833646 --- Validating RPKI --- Validating RPKI ROAs 208232 raw RKPI ROA files found. 208232 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1702994400/rpki/rpki_raw.json ...finished in 0:33:34.561066 --- Parsing RPKI --- Parsing 208232 ROAs Result entries written: 363070 Duplicates found: 34173 Invalids found: 111766 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:40.755001 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/lacnic.db Found valid entries: 12216 Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/ripe.db.route Found valid entries: 405430 Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/apnic.db.route6 Found valid entries: 328726 Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/apnic.db.route Found valid entries: 692914 Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/arin.db Found valid entries: 102228 Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/afrinic.db Found valid entries: 112301 Parsing /Users/FJ/projects/python/kartograf/out/1702994400/irr/ripe.db.route6 Found valid entries: 197056 ...finished in 0:03:14.493636 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 185079 / 185079 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185079 / 185079 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185079 / 185079 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 185078 / 185078 | Finished filtering! Originally 1850783 entries filtered down to 559052 Merging base file with filtered extra file ...finished in 0:53:18.926362 --- Parsing Routeviews pfx2as --- Unzipping /Users/FJ/projects/python/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/FJ/projects/python/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/FJ/projects/python/kartograf/out/1702994400/collectors/pfx2asn.txt ...finished in 0:01:55.281757 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 118879 / 118879 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118879 / 118879 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 118878 / 118878 | Finished filtering! Originally 1188782 entries filtered down to 427304 Merging base file with filtered extra file ...finished in 1:41:49.021772 --- Sorting results --- ...finished in 0:00:07.812508 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 0df6aa452452d964f10ae2100e094f96e142051ff817fb75fabb90cdbdfd2bbf Total runtime: 3:23:48.054868 ```

fjahr commented 10 months ago

It looks like it's the ARIN certificate again, my logs say it couldn't find the certificate. Though somehow, magically, @Sjors computer always has it ready to go 🙄

brunoerg commented 10 months ago

Could you both @fjahr and @Sjors share your machine specs? Here is taking 3x more time to complete.

fjahr commented 10 months ago

@brunoerg It's most certainly the CPU heavy tasks. The library I use for multithreading automatically adjusts to what it has available, you can see this in the number of progress bars in the logs: you have 8, I have 10, Sjors has 16. So the number of available cores is one factor. You mentioned you have an M1 Macbook but not Pro, right? I have an M1 MBP so probably my M1 is a bit more powerful than yours. Also am I right that the normal MacBooks are passively cooled? It might be that you experience throttling to prevent overheating, my MBP is running the fans on high speed almost the whole time when Kartograf is running. Does that sound about right or did you have a MBP, then we might want to dig deeper?

0xB10C commented 10 months ago

The SHA-256 hash of the result file is: d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014
Total runtime: 6:02:32.218789

edit: this machine is also doing a mainnet IBD currently. Might have affected available bandwidth and CPU.

``` --- Start Kartograf --- Using rpki-client version 8.6. Coordinated launch mode: Waiting until 1702994400 (2023-12-19 15:00:00 CET) to launch mapping process. The epoch for this run is: 1702994400 (2023-12-19 14:00:00 UTC, local: 2023-12-19 15:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/b10c/kartograf/data/1702994400/rpki/tals/afrinic.tal Downloaded TAL for APNIC to /home/b10c/kartograf/data/1702994400/rpki/tals/apnic.tal Downloaded TAL for ARIN to /home/b10c/kartograf/data/1702994400/rpki/tals/arin.tal Downloaded TAL for LACNIC to /home/b10c/kartograf/data/1702994400/rpki/tals/lacnic.tal Downloaded TAL for RIPE to /home/b10c/kartograf/data/1702994400/rpki/tals/ripe.tal Downloading RPKI Data ...finished in 0:03:06.320873 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:00:36.206011 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231217-1200.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231218-1200.pfx2as.gz ...finished in 0:00:07.093809 --- Validating RPKI --- Validating RPKI ROAs 208231 raw RKPI ROA files found. 208231 RKPI ROAs validated and saved to /home/b10c/kartograf/out/1702994400/rpki/rpki_raw.json ...finished in 0:19:03.543005 --- Parsing RPKI --- Parsing 208231 ROAs Result entries written: 363079 Duplicates found: 34174 Invalids found: 111755 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:09.533552 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/b10c/kartograf/out/1702994400/irr/afrinic.db Found valid entries: 112301 Parsing /home/b10c/kartograf/out/1702994400/irr/apnic.db.route Found valid entries: 692914 Parsing /home/b10c/kartograf/out/1702994400/irr/apnic.db.route6 Found valid entries: 328726 Parsing /home/b10c/kartograf/out/1702994400/irr/arin.db Found valid entries: 102228 Parsing /home/b10c/kartograf/out/1702994400/irr/lacnic.db Found valid entries: 12216 Parsing /home/b10c/kartograf/out/1702994400/irr/ripe.db.route Found valid entries: 405430 Parsing /home/b10c/kartograf/out/1702994400/irr/ripe.db.route6 Found valid entries: 197056 ...finished in 0:05:35.685249 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 308464 / 308464 | ... 100.00% :::::::::::::::::::::::::::::::::::::::: | 308463 / 308463 | Finished filtering! Originally 1850783 entries filtered down to 559022 Merging base file with filtered extra file ...finished in 1:58:57.976723 --- Parsing Routeviews pfx2as --- Unzipping /home/b10c/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/b10c/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/b10c/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/b10c/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/b10c/kartograf/out/1702994400/collectors/pfx2asn.txt ...finished in 0:03:15.747656 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 198130 / 198130 | ... 100.00% :::::::::::::::::::::::::::::::::::::::: | 198130 / 198130 | Finished filtering! Originally 1188782 entries filtered down to 427303 Merging base file with filtered extra file ...finished in 3:30:25.562101 --- Sorting results --- ...finished in 0:00:12.860962 --- Finishing Kartograf --- The SHA-256 hash of the result file is: d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014 Total runtime: 6:02:32.218789 ```

Sjors commented 10 months ago

@brunoerg AMD Ryzen 7950X with 16 cores / 32 threads, 64 GB RAM.

jurraca commented 10 months ago

The SHA-256 hash of the result file is: d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014
Total runtime: 8:50:02.826999

It only used half my cores, won't make that mistake again hah.

``` --- Start Kartograf --- Using rpki-client version 8.6. Coordinated launch mode: Waiting until 1702994400 (2023-12-19 14:00:00 WET) to launch mapping process. The epoch for this run is: 1702994400 (2023-12-19 14:00:00 UTC, local: 2023-12-19 14:00:00 WET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/base/code/kartograf/data/1702994400/rpki/tals/afrinic.tal Downloaded TAL for APNIC to /home/base/code/kartograf/data/1702994400/rpki/tals/apnic.tal Downloaded TAL for ARIN to /home/base/code/kartograf/data/1702994400/rpki/tals/arin.tal Downloaded TAL for LACNIC to /home/base/code/kartograf/data/1702994400/rpki/tals/lacnic.tal Downloaded TAL for RIPE to /home/base/code/kartograf/data/1702994400/rpki/tals/ripe.tal Downloading RPKI Data ...finished in 0:03:49.174983 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:00:59.567449 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231217-1200.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231218-1200.pfx2as.gz ...finished in 0:00:10.034538 --- Validating RPKI --- --- Validating RPKI --- Validating RPKI ROAs 208232 raw RKPI ROA files found. 208232 RKPI ROAs validated and saved to /home/base/code/kartograf/out/1702994400/rpki/rpki_raw.json ...finished in 0:24:18.490872 --- Parsing RPKI --- Parsing 208232 ROAs Result entries written: 363080 Duplicates found: 34174 Invalids found: 111755 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:18.259234 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/base/code/kartograf/out/1702994400/irr/afrinic.db Found valid entries: 112301 Parsing /home/base/code/kartograf/out/1702994400/irr/arin.db Found valid entries: 102228 Parsing /home/base/code/kartograf/out/1702994400/irr/ripe.db.route6 Found valid entries: 197056 Parsing /home/base/code/kartograf/out/1702994400/irr/apnic.db.route6 Found valid entries: 328726 Parsing /home/base/code/kartograf/out/1702994400/irr/lacnic.db Found valid entries: 12216 Parsing /home/base/code/kartograf/out/1702994400/irr/lacnic.db [35/201] Found valid entries: 12216 Parsing /home/base/code/kartograf/out/1702994400/irr/ripe.db.route Found valid entries: 405430 Parsing /home/base/code/kartograf/out/1702994400/irr/apnic.db.route Found valid entries: 692914 ...finished in 0:06:17.401052 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 3.63% : 3.63% : | 8393 / 231348 | 3.63% : | 8398 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 231347 / 231347 | Finished filtering! Originally 1850783 entries filtered down to 559021 Merging base file with filtered extra file ...finished in 3:00:52.205440 --- Parsing Routeviews pfx2as --- Unzipping /home/base/code/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/base/code/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/base/code/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/base/code/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/base/code/kartograf/out/1702994400/collectors/pfx2asn.txt ...finished in 0:03:46.518016 Cleaning /home/base/code/kartograf/out/1702994400/collectors/pfx2asn.txt [0/201] ...finished in 0:03:46.518016 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 4.85% 4.85% : | 7212 / 148598 | % : 4.85% : | 7213 / 148598 | 4.85% : 4.86% : | 7216 / 148598 | 4.86% : | 7217 / 148598 | 4.86% : | 7217 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 148597 / 148597 | ^ 100.00% :::::::::::::::::::::::::::::::::::::::: | 148597 / 148597 | Finished filtering! Originally 1188782 entries filtered down to 427303 Merging base file with filtered extra file ...finished in 5:07:07.864647 --- Sorting results --- ...finished in 0:00:12.807423 --- Finishing Kartograf --- The SHA-256 hash of the result file is: d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014 Total runtime: 8:50:02.826999 ```

brunoerg commented 10 months ago

The SHA-256 hash of the result file is: d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014
Total runtime: 20:56:18.538361

Details

--- Start Kartograf --- Using rpki-client version 8.6. Coordinated launch mode: Waiting until 1702994400 (2023-12-19 11:00:00 -03) to launch mapping process. The epoch for this run is: 1702994400 (2023-12-19 14:00:00 UTC, local: 2023-12-19 11:00:00 -03) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/brunogarcia/projects/kartograf/data/1702994400/rpki/tals/afrinic.tal Downloaded TAL for APNIC to /Users/brunogarcia/projects/kartograf/data/1702994400/rpki/tals/apnic.tal Downloaded TAL for ARIN to /Users/brunogarcia/projects/kartograf/data/1702994400/rpki/tals/arin.tal Downloaded TAL for LACNIC to /Users/brunogarcia/projects/kartograf/data/1702994400/rpki/tals/lacnic.tal Downloaded TAL for RIPE to /Users/brunogarcia/projects/kartograf/data/1702994400/rpki/tals/ripe.tal Downloading RPKI Data ...finished in 0:05:54.567778 --- Fetching IRR --- Downloading afrinic.db.gz Downloading apnic.db.route.gz Downloading apnic.db.route6.gz Downloading arin.db.gz Downloading lacnic.db.gz Downloading ripe.db.route.gz Downloading ripe.db.route6.gz ...finished in 0:01:38.990768 --- Fetching Routeviews pfx2as --- Downloading https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2023/12/routeviews-rv2-20231217-1200.pfx2as.gz Downloading https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2023/12/routeviews-rv6-20231218-1200.pfx2as.gz ...finished in 0:00:08.591438 --- Validating RPKI --- Validating RPKI ROAs 208231 raw RKPI ROA files found. 208231 RKPI ROAs validated and saved to /Users/brunogarcia/projects/kartograf/out/1702994400/rpki/rpki_raw.json ...finished in 1:03:29.636653 --- Parsing RPKI --- Parsing 208231 ROAs Result entries written: 363079 Duplicates found: 34174 Invalids found: 111755 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:50.333508 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/lacnic.db Found valid entries: 12216 Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/ripe.db.route Found valid entries: 405430 Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/apnic.db.route6 Found valid entries: 328726 Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/apnic.db.route Found valid entries: 692914 Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/arin.db Found valid entries: 102228 Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/afrinic.db Found valid entries: 112301 Parsing /Users/brunogarcia/projects/kartograf/out/1702994400/irr/ripe.db.route6 Found valid entries: 197056 ...finished in 0:03:48.403135 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231348 / 231348 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 231347 / 231347 | Finished filtering! Originally 1850783 entries filtered down to 559022 Merging base file with filtered extra file ...finished in 3:13:18.675915 --- Parsing Routeviews pfx2as --- Unzipping /Users/brunogarcia/projects/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/brunogarcia/projects/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/brunogarcia/projects/kartograf/data/1702994400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/brunogarcia/projects/kartograf/out/1702994400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/brunogarcia/projects/kartograf/out/1702994400/collectors/pfx2asn.txt ...finished in 0:02:05.318481 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 7.01% :: | 10423 / 148598 | 7.03% :: | 10451 / 148598 | 6.47% :: | 9614 / 148598 | 6.48% :: | 9628 / 148598 | 6.56% :: | 9743 / 148598 | 6.56% :: | 9741 / 148598 | 7.02% :: | 10427 / 148598 | 7.02% :: | 10433 / 148598 | 64 64.47% ::::::::::::::::::::::::: | 95800 / 148598 | 64.5 64.56% ::::::::::::::::::::::::: | 95942 / 148598 | 64.65% 64.66% ::::::::::::::::::::::::: | 96090 / 148598 | 64.76% : 64.78% ::::::::::::::::::::::::: | 96264 / 148598 | 65.01% ::: 65.03% :::::::::::::::::::::::::: | 96638 / 148598 | 65.53% ::::: 65.55% :::::::::::::::::::::::::: | 97402 / 148598 | 65.84% ::::::: 65.86% :::::::::::::::::::::::::: | 97860 / 148598 | 66.13% ::::::::: 66.15% :::::::::::::::::::::::::: | 98305 / 148598 | 66.46% ::::::::::: 66.48% :::::::::::::::::::::::::: | 98784 / 148598 | 67.60% ::::::::::::: 67.62% ::::::::::::::::::::::::::: | 100477 / 148598 | 67.78% ::::::::::::::: 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148598 / 148598 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148597 / 148597 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 148597 / 148597 | Finished filtering! Originally 1188782 entries filtered down to 427303 Merging base file with filtered extra file ...finished in 4:05:41.042185 --- Sorting results --- ...finished in 0:00:06.413932 --- Finishing Kartograf --- The SHA-256 hash of the result file is: d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014 Total runtime: 20:56:18.538361

0xB10C commented 10 months ago

Seems like:

1x fac1322ab662097f3d9528b8a885a77589850fd1d73097007198a89edbce436a @Sjors
1x 0df6aa452452d964f10ae2100e094f96e142051ff817fb75fabb90cdbdfd2bbf @fjahr
3x d3d9144088e95143cd988145099a5224a0ba2fd48a65ce31772dccfa8f148014 @jurraca @brunoerg @0xB10C

Should we do another run?

jurraca commented 10 months ago

maybe we should diff the final_result.txt files we got to see the discrepancy?

0xB10C commented 10 months ago

Mine is here: final_result.txt.gz (4.7 MB)

Sjors commented 10 months ago

Here you go: final_result.txt.gz (4.9 MB)

fjahr commented 10 months ago

maybe we should diff the final_result.txt files we got to see the discrepancy?

What will be interesting to look at is the difference between mine and the one that you got. @Sjors will have a quite different result because of the same issue I described above. I thought I had fixed it but I didn't really. But I have found it now and pushed a release with the fix just now. See also the conversation at rpki-client where they cleared up my misunderstanding of how the cache dir works: https://github.com/rpki-client/rpki-client-portable/issues/101

So @Sjors is the only one with a real, full result. But the other four are at least all incomplete in the same way, so the diff between me and the rest is something that we might also see when the bug is fixed, so it might still be interesting to look at.

Here is my result file: final_result.txt.zip

I would like to do another run tomorrow at the time the IRC meeting starts (2pm UTC), would be great if some of you could join me again. I am confident that the last issue is now fully fixed. Timestamp: 1703167200 And for this please use Kartograf 0.3.2: https://github.com/fjahr/kartograf/releases/tag/0.3.2

./run map -w=1703167200 -irr -rv

Thanks a lot for your patience ❤️ and let me know if you have questions!

Sjors commented 10 months ago

I think you meant 2pm UTC and 3pm CET, which is also what the timestamp says.

asmap / asmap-data

Demo using Collaborative Launch feature at 1702994400 (Kartograf 0.4.1) #4