search

asmap / asmap-data

Demo repository for how a similar repository could be used in Bitcoin Core
MIT License
6 stars 2 forks source link

Collaborative Launch 1724248800 (Wednesday, August 21, 2024 2:00 PM GMT) #15

Closed fjahr closed 3 months ago

fjahr commented 3 months ago

Next collaborative launch is planned for 1724248800. Please make sure to use Kartograf v0.4.6 and this time please also try to use the latest rpki-client version. The run will work with an older version, too, but I would like to test if our success rate can be improved if everyone is one the latest version. This could hint at "silent conflicts" produced by different versions of rpki-client. See here for more info on this: https://github.com/fjahr/kartograf/issues/16.

./run map -w=1724248800 -irr -rv

EDIT: nix users please use kartograf 0.4.7. This ensures you will be on the latest rpki-client version.

0xB10C commented 3 months ago

fwiw: https://github.com/fjahr/rpki-client-nix/issues/7

Sjors commented 3 months ago

Count me in!

It might be useful to have a ./run check which does some sanity checking and spits out relevant info like the rpki-client version.

I'm at 9.0, which is the latest shipped with Ubuntu: https://launchpad.net/ubuntu/noble/+package/rpki-client

I'll try building 9.1 from source.

Sjors commented 3 months ago

If anyone I know feels like signing this key... https://ftp.openbsd.org/pub/OpenBSD/rpki-client/RELEASE_KEY.asc

Mmm, building from source on Ubuntu seems very non-trivial, since you need LibreSSL / libtls? I'll try the nix route (but that's not ready yet).

fjahr commented 3 months ago

which is the latest shipped with Ubuntu

Oh, weird, on 24.10 it seems 9.1 is available but not on 24.04: https://launchpad.net/ubuntu/+source/rpki-client But I don't know enough about the Ubuntu packages if that is unusual.

I'll try the nix route (but that's not ready yet).

We'll try to have it ready by Wednesday.

brunoerg commented 3 months ago

I'm in! Just updated rpki-client to 9.1 on MacOS.

Kartograf version: 0.4.6
Using rpki-client version 9.1 (recommended).
Coordinated launch mode: Waiting until 1724248800 (2024-08-21 11:00:00 -03) to launch mapping process.
Countdown: 1 day(s), 19 hour(s), 13 minute(s), 26 second(s)
Sjors commented 3 months ago

Ubuntu 24.10 won't be released until October. I'm running 24.04.

Sjors commented 3 months ago

I'm in! Just updated rpki-client to 9.1 on MacOS.

I can do it on macOS too if if nix isn't ready on time.

Sjors commented 3 months ago

I just tried it twice between macOS with 9.1 and Ubuntu with 9.0 for the same timestamps. Both times I ended up with different RPKI Data, hash sum. Could that be due to the version difference? Or perhaps due to wildly different download speed (2 minutes on AMD Ryzen 7950x, 12 minutes on 2019 Intel MPB, the same LAN).

fjahr commented 3 months ago

I just tried it twice between macOS with 9.1 and Ubuntu with 9.0 for the same timestamps. Both times I ended up with different RPKI Data, hash sum. Could that be due to the version difference? Or perhaps due to wildly different download speed (2 minutes on AMD Ryzen 7950x, 12 minutes on 2019 Intel MPB, the same LAN).

I assume you started both at exactly the same timestamp using the -w flag? Then the download speed would be one explanation or it's one of these "silent conflicts" that I mentioned, so the version difference.

If you produce a map from the just rpki data it would be interesting to see what the diff is. Or you could look at the file tree to see if the difference there is small enough. It's the annoying part of asmap that debugging these things are often pretty time intensive. If you could share your raw rpki data from both macos and ubuntu with the same timestamp then I could try to take a look but I probably won't get it done before we do our run here.

Last but not least, since there is a massive de-dublication of the RPKI data the end result can still be a match. Did you finish the process and receive a different end result as well?

fjahr commented 3 months ago

fwiw: fjahr/rpki-client-nix#7

This should be fixed now. nix users please use kartograf 0.4.7, for non-nix users there is no difference to 0.4.6.

0xB10C commented 3 months ago

This should be fixed now. nix users please use kartograf 0.4.7, for non-nix users there is no difference to 0.4.6.

Thanks!

Using rpki-client version 9.1 (recommended).
Sjors commented 3 months ago

I assume you started both at exactly the same timestamp using the -w flag?

Yes

Did you finish the process and receive a different end result as well?

No

but I probably won't get it done before we do our run here.

Let's just wait and see how it goes tomorrow, and then upload stuff where needed.

jurraca commented 3 months ago

T-29 minutes til takeoff

hebasto commented 3 months ago 
$ ./run map -w=1724248800 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.6
Using rpki-client version 9.0. Please beware that running with the latest tested version (9.1) is recommend.
Coordinated launch mode: Waiting until 1724248800 (2024-08-21 15:00:00 BST) to launch mapping process.
...
jurraca commented 3 months ago

my run errored out with OSError: AF_UNIX path too long in the merging RPKI and IRR step. Tried rerunning but it didn't finish downloading TALs, so can't rerun the same run. Looking into it.

fjahr commented 3 months ago

I finished with the final hash of 4474291b341d9b36a2fb743bfb6c335409d56a2a6d6a184c2c4e13184acaecd2.

Full logs ``` $ [nix] ./run map -w=1724248800 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.7 Using rpki-client version 9.1 (recommended). Coordinated launch mode: Waiting until 1724248800 (2024-08-21 16:00:00 CEST) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1724248800 (2024-08-21 14:00:00 UTC, local: 2024-08-21 16:00:00 CEST) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1724248800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1724248800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1724248800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1724248800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1724248800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: d408d90b43237ace7a89a4c0995e6f9138f967f79a75dbd84be336120239a3df ...finished in 0:04:00.489454 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 4e2636c48738f81a244864c39f1afaead841305b10313d2cab4415bd145b918d Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: dcf3a00a94cdb4db6b8344f1d4e1e53215b2d5ddc320b1f8c84791cd73f63d6d Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 61c92d5cc0118b5928fb5eaf47e2c27f1c6d9ef49653be6045ab8ec85019ecb3 Downloading arin.db.gz Downloaded arin.db.gz, file hash: cadb285919d60b1652bf1048c25564163991c584bfcfba2c20524b0271b2df9c Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: 9cf49c26a48c1688d8674abeda8cc94c594d9fa180e8333ec46f562384b5299c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 714350fd59c119da833c5a0da515cee0f93de2b476095e9bae4993b9f85096bb Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 3cf7f061439d9932f46ce1a5c9fbae6d347f4736593fcf1a5fa7dbe5494ab8ee ...finished in 0:01:01.818340 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/08/routeviews-rv2-20240819-1200.pfx2as.gz Downloaded /Users/FJ/projects/python/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 1b8827c21718ff44552534ad720b4ae323926a0e89891adefc9de298427e848d Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/08/routeviews-rv6-20240820-1200.pfx2as.gz Downloaded /Users/FJ/projects/python/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 13f16d712e4451aba3b70ab2292dbf711322112004f902724b5c95d19b24ba4c ...finished in 0:00:07.307448 --- Validating RPKI --- Validating RPKI ROAs 163900 raw RKPI ROA files found. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 163900/163900 [25:11<00:00, 108.41it/s] 163900 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1724248800/rpki/rpki_raw.json, file hash: 9355180b2191720a57bb5210dae107a9a6990c318bf75736f1c6ab9237be9c0e ...finished in 0:26:05.713264 --- Parsing RPKI --- Parsing 163900 ROAs Result entries written: 401562 Duplicates found: 40055 Invalids found: 51471 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:55.113851 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/lacnic.db Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/ripe.db.route Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/apnic.db.route6 Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/apnic.db.route Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/arin.db Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/afrinic.db Parsing /Users/FJ/projects/python/kartograf/out/1724248800/irr/ripe.db.route6 Found valid, unique entries: 1787236 ...finished in 0:04:44.078339 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | Finished filtering! Originally 1787236 entries filtered down to 411803 Merging base file with filtered extra file ...finished in 0:58:44.895492 --- Parsing Routeviews pfx2as --- Unzipping /Users/FJ/projects/python/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/FJ/projects/python/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/FJ/projects/python/kartograf/out/1724248800/collectors/pfx2asn.txt ...finished in 0:02:24.884397 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | Finished filtering! Originally 1227815 entries filtered down to 472265 Merging base file with filtered extra file ...finished in 1:19:20.534416 --- Sorting results --- ...finished in 0:00:13.524074 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 4474291b341d9b36a2fb743bfb6c335409d56a2a6d6a184c2c4e13184acaecd2 Total runtime: 2:57:41.033361 ```
brunoerg commented 3 months ago

Just finished it with the final hash of 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153.

Logs
--- Start Kartograf --- Kartograf version: 0.4.6 Using rpki-client version 9.1 (recommended). Coordinated launch mode: Waiting until 1724248800 (2024-08-21 11:00:00 -03) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1724248800 (2024-08-21 14:00:00 UTC, local: 2024-08-21 11:00:00 -03) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/brunogarcia/projects/kartograf/data/1724248800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /Users/brunogarcia/projects/kartograf/data/1724248800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /Users/brunogarcia/projects/kartograf/data/1724248800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /Users/brunogarcia/projects/kartograf/data/1724248800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /Users/brunogarcia/projects/kartograf/data/1724248800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 6d56d8da06b26d5b8409b42c8ad912e05e2287601eb1766846dc97a7580a51c3 ...finished in 0:06:21.263750 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 4e2636c48738f81a244864c39f1afaead841305b10313d2cab4415bd145b918d Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: dcf3a00a94cdb4db6b8344f1d4e1e53215b2d5ddc320b1f8c84791cd73f63d6d Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 61c92d5cc0118b5928fb5eaf47e2c27f1c6d9ef49653be6045ab8ec85019ecb3 Downloading arin.db.gz Downloaded arin.db.gz, file hash: cadb285919d60b1652bf1048c25564163991c584bfcfba2c20524b0271b2df9c Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: 9cf49c26a48c1688d8674abeda8cc94c594d9fa180e8333ec46f562384b5299c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 714350fd59c119da833c5a0da515cee0f93de2b476095e9bae4993b9f85096bb Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 3cf7f061439d9932f46ce1a5c9fbae6d347f4736593fcf1a5fa7dbe5494ab8ee ...finished in 0:01:24.574852 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/08/routeviews-rv2-20240819-1200.pfx2as.gz Downloaded /Users/brunogarcia/projects/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 1b8827c21718ff44552534ad720b4ae323926a0e89891adefc9de298427e848d Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/08/routeviews-rv6-20240820-1200.pfx2as.gz Downloaded /Users/brunogarcia/projects/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 13f16d712e4451aba3b70ab2292dbf711322112004f902724b5c95d19b24ba4c ...finished in 0:00:08.206793 --- Validating RPKI --- Validating RPKI ROAs 310253 raw RKPI ROA files found. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 310253/310253 [25:12<00:00, 205.08it/s] 310253 RKPI ROAs validated and saved to /Users/brunogarcia/projects/kartograf/out/1724248800/rpki/rpki_raw.json, file hash: 9c7473bdc772e42b8b731b32d98b79757d369e57b5b2fb9e78abb9e1622b9932 ...finished in 0:25:33.515148 --- Parsing RPKI --- Parsing 310253 ROAs Result entries written: 543064 Duplicates found: 66213 Invalids found: 49997 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:54.144818 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/lacnic.db Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/ripe.db.route Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/apnic.db.route6 Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/apnic.db.route Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/arin.db Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/afrinic.db Parsing /Users/brunogarcia/projects/kartograf/out/1724248800/irr/ripe.db.route6 Found valid, unique entries: 1787236 ...finished in 0:03:12.657127 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 96.98% :::::::::::::::::::::::::::::::::::::: | 173318 / 178724 | 97.47% :::::::::::::::::::::::::::::::::::::: | 174207 / 178724 | 91.51% :::::::::::::::::::::::::::::::::::: | 163544 / 178724 | 90.33% :::::::::::::::::::::::::::::::::::: | 161442 / 178724 | 88.32% ::::::::::::::::::::::::::::::::::: | 157854 / 178724 | 96.98% :::::::::::::::::::::::::::::::::::::: | 173329 / 178724 | | 155699 / 178724 | 97.48% :::::::::::::::::::::::::::::::::::::: | 174218 / 178724 | | 157701 / 178723 | 91.51% :::::::::::::::::::::::::::::::::::: | 163553 / 178724 | :: | 171802 / 178723 | 90.34% :::::::::::::::::::::::::::::::::::: | 161451 / 178724 | | 151891 / 178723 | 88.33% ::::::::::::::::::::::::::::::::::: | 157864 / 178724 | 96.98% :::::::::::::::::::::::::::::::::::::: | 173329 / 178724 | 87.12% :::::::::::::::::::::::::::::::::: | 155699 / 178724 | 97.48% :::::::::::::::::::::::::::::::::::::: | 174228 / 178724 | 91.52% :::::::::::::::::::::::::::::::::::: | 163560 / 178724 | 90.34% :::::::::::::::::::::::::::::::::::: | 161459 / 178724 | 88.33% ::::::::::::::::::::::::::::::::::: | 157873 / 178724 | 96.99% :::::::::::::::::::::::::::::::::::::: | 173340 / 178724 | 97.49% :::::::::::::::::::::::::::::::::::::: | 174238 / 178724 | 91.52% :::::::::::::::::::::::::::::::::::: | 163567 / 178724 | 90.34% :::::::::::::::::::::::::::::::::::: | 161466 / 178724 | 88.34% ::::::::::::::::::::::::::::::::::: | 157880 / 178724 | 96.99% :::::::::::::::::::::::::::::::::::::: | 173349 / 178724 | 97.50% :::::::::::::::::::::::::::::::::::::: | 174248 / 178724 | 91.52% :::::::::::::::::::::::::::::::::::: | 163575 / 178724 | 90.35% :::::::::::::::::::::::::::::::::::: | 161474 / 178724 | 88.34% ::::::::::::::::::::::::::::::::::: | 157888 / 178724 | 97.00% :::::::::::::::::::::::::::::::::::::: | 173358 / 178724 | 97.50% ::::::::::::::::::::::::::::::::::::::: | 174258 / 178724 | 91.53% :::::::::::::::::::::::::::::::::::: | 163586 / 178724 | 90.35% :::::::::::::::::::::::::::::::::::: | 161483 / 178724 | 88.35% ::::::::::::::::::::::::::::::::::: | 157897 / 178724 | 97.01% :::::::::::::::::::::::::::::::::::::: | 173380 / 178724 | 97.51% ::::::::::::::::::::::::::::::::::::::: | 174268 / 178724 | 91.54% :::::::::::::::::::::::::::::::::::: | 163596 / 178724 | 90.36% :::::::::::::::::::::::::::::::::::: | 161492 / 178724 | 88.35% ::::::::::::::::::::::::::::::::::: | 157907 / 178724 | 97.01% :::::::::::::::::::::::::::::::::::::: | 173380 / 178724 | 97.51% ::::::::::::::::::::::::::::::::::::::: | 174277 / 178724 | 91.54% :::::::::::::::::::::::::::::::::::: | 163606 / 178724 | 90.36% :::::::::::::::::::::::::::::::::::: | 161501 / 178724 | 88.36% ::::::::::::::::::::::::::::::::::: | 157915 / 178724 | 97.02% :::::::::::::::::::::::::::::::::::::: | 173390 / 178724 | 97.52% ::::::::::::::::::::::::::::::::::::::: | 174287 / 178724 | 91.55% :::::::::::::::::::::::::::::::::::: | 163615 / 178724 | 90.37% :::::::::::::::::::::::::::::::::::: | 161508 / 178724 | 88.36% ::::::::::::::::::::::::::::::::::: | 157923 / 178724 | 97.02% :::::::::::::::::::::::::::::::::::::: | 173401 / 178724 | 97.52% ::::::::::::::::::::::::::::::::::::::: | 174298 / 178724 | 91.55% :::::::::::::::::::::::::::::::::::: | 163625 / 178724 | 90.37% :::::::::::::::::::::::::::::::::::: | 161517 / 178724 | 88.37% ::::::::::::::::::::::::::::::::::: | 157933 / 178724 | 97.03% :::::::::::::::::::::::::::::::::::::: | 173412 / 178724 | 97.53% ::::::::::::::::::::::::::::::::::::::: | 174309 / 178724 | 91.56% :::::::::::::::::::::::::::::::::::: | 163634 / 178724 | 90.38% :::::::::::::::::::::::::::::::::::: | 161525 / 178724 | 88.37% ::::::::::::::::::::::::::::::::::: | 157941 / 178724 | 97.03% :::::::::::::::::::::::::::::::::::::: | 173423 / 178724 | 97.54% ::::::::::::::::::::::::::::::::::::::: | 174320 / 178724 | 91.56% :::::::::::::::::::::::::::::::::::: | 163643 / 178724 | 90.38% :::::::::::::::::::::::::::::::::::: | 161533 / 178724 | 88.38% ::::::::::::::::::::::::::::::::::: | 157950 / 178724 | 97.05% :::::::::::::::::::::::::::::::::::::: | 173444 / 178724 | 97.54% ::::::::::::::::::::::::::::::::::::::: | 174330 / 178724 | 91.57% :::::::::::::::::::::::::::::::::::: | 163652 / 178724 | 90.39% :::::::::::::::::::::::::::::::::::: | 161542 / 178724 | 88.38% ::::::::::::::::::::::::::::::::::: | 157960 / 178724 | 97.05% :::::::::::::::::::::::::::::::::::::: | 173444 / 178724 | 97.55% ::::::::::::::::::::::::::::::::::::::: | 174340 / 178724 | 91.57% :::::::::::::::::::::::::::::::::::: | 163662 / 178724 | 90.39% :::::::::::::::::::::::::::::::::::: | 161551 / 178724 | 88.39% ::::::::::::::::::::::::::::::::::: | 157970 / 178724 | 97.05% :::::::::::::::::::::::::::::::::::::: | 173455 / 178724 | 97.55% ::::::::::::::::::::::::::::::::::::::: | 174350 / 178724 | 91.58% :::::::::::::::::::::::::::::::::::: | 163671 / 178724 | 90.40% :::::::::::::::::::::::::::::::::::: | 161569 / 178724 | 88.39% ::::::::::::::::::::::::::::::::::: | 157979 / 178724 | 97.06% :::::::::::::::::::::::::::::::::::::: | 173476 / 178724 | 97.56% ::::::::::::::::::::::::::::::::::::::: | 174360 / 178724 | 91.58% :::::::::::::::::::::::::::::::::::: | 163679 / 178724 | 90.40% :::::::::::::::::::::::::::::::::::: | 161569 / 178724 | 88.40% ::::::::::::::::::::::::::::::::::: | 157987 / 178724 | 97.07% :::::::::::::::::::::::::::::::::::::: | 173487 / 178724 | 97.56% ::::::::::::::::::::::::::::::::::::::: | 174370 / 178724 | 91.59% :::::::::::::::::::::::::::::::::::: | 163688 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178724 / 178724 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 178723 / 178723 | Finished filtering! Originally 1787236 entries filtered down to 378735 Merging base file with filtered extra file ...finished in 1:13:50.465611 --- Parsing Routeviews pfx2as --- Unzipping /Users/brunogarcia/projects/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/brunogarcia/projects/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/brunogarcia/projects/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/brunogarcia/projects/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/brunogarcia/projects/kartograf/out/1724248800/collectors/pfx2asn.txt ...finished in 0:01:41.098785 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122782 / 122782 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 122781 / 122781 | Finished filtering! Originally 1227815 entries filtered down to 386230 Merging base file with filtered extra file ...finished in 1:25:36.066030 --- Sorting results --- ...finished in 0:00:09.841526 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 Total runtime: 3:18:53.000116
Sjors commented 3 months ago

0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153

Log ``` $ ./run map -w=1724248800 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.7 Using rpki-client version 9.1 (recommended). Coordinated launch mode: Waiting until 1724248800 (2024-08-21 16:00:00 CEST) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1724248800 (2024-08-21 14:00:00 UTC, local: 2024-08-21 16:00:00 CEST) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/sjors/dev/kartograf/data/1724248800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /home/sjors/dev/kartograf/data/1724248800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /home/sjors/dev/kartograf/data/1724248800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /home/sjors/dev/kartograf/data/1724248800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /home/sjors/dev/kartograf/data/1724248800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 3acd465392f3a315f8ac4718bdc995032cd62e2c7b3da43ad141e073f5e6ec21 ...finished in 0:02:59.503657 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 4e2636c48738f81a244864c39f1afaead841305b10313d2cab4415bd145b918d Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: dcf3a00a94cdb4db6b8344f1d4e1e53215b2d5ddc320b1f8c84791cd73f63d6d Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 61c92d5cc0118b5928fb5eaf47e2c27f1c6d9ef49653be6045ab8ec85019ecb3 Downloading arin.db.gz Downloaded arin.db.gz, file hash: cadb285919d60b1652bf1048c25564163991c584bfcfba2c20524b0271b2df9c Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: 9cf49c26a48c1688d8674abeda8cc94c594d9fa180e8333ec46f562384b5299c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 714350fd59c119da833c5a0da515cee0f93de2b476095e9bae4993b9f85096bb Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 3cf7f061439d9932f46ce1a5c9fbae6d347f4736593fcf1a5fa7dbe5494ab8ee ...finished in 0:00:35.449542 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/08/routeviews-rv2-20240819-1200.pfx2as.gz Downloaded /home/sjors/dev/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 1b8827c21718ff44552534ad720b4ae323926a0e89891adefc9de298427e848d Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/08/routeviews-rv6-20240820-1200.pfx2as.gz Downloaded /home/sjors/dev/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 13f16d712e4451aba3b70ab2292dbf711322112004f902724b5c95d19b24ba4c ...finished in 0:00:07.718679 --- Validating RPKI --- Validating RPKI ROAs 310253 raw RKPI ROA files found. 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 310253/310253 [05:42<00:00, 905.21it/s] 310253 RKPI ROAs validated and saved to /home/sjors/dev/kartograf/out/1724248800/rpki/rpki_raw.json, file hash: 37a5498051bfa76c017ebde6e2d4ed13ac8ffa38707bfe5ecf2fae49558fc1ed ...finished in 0:05:53.501008 --- Parsing RPKI --- Parsing 310253 ROAs Result entries written: 543064 Duplicates found: 66213 Invalids found: 49997 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:50.373575 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/sjors/dev/kartograf/out/1724248800/irr/arin.db Parsing /home/sjors/dev/kartograf/out/1724248800/irr/afrinic.db Parsing /home/sjors/dev/kartograf/out/1724248800/irr/lacnic.db Parsing /home/sjors/dev/kartograf/out/1724248800/irr/ripe.db.route Parsing /home/sjors/dev/kartograf/out/1724248800/irr/apnic.db.route6 Parsing /home/sjors/dev/kartograf/out/1724248800/irr/ripe.db.route6 Parsing /home/sjors/dev/kartograf/out/1724248800/irr/apnic.db.route Found valid, unique entries: 1787236 ...finished in 0:03:11.416269 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | Finished filtering! Originally 1787236 entries filtered down to 378735 Merging base file with filtered extra file ...finished in 1:18:30.796988 --- Parsing Routeviews pfx2as --- Unzipping /home/sjors/dev/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/sjors/dev/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/sjors/dev/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/sjors/dev/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/sjors/dev/kartograf/out/1724248800/collectors/pfx2asn.txt ...finished in 0:01:39.571856 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | Finished filtering! Originally 1227815 entries filtered down to 386230 Merging base file with filtered extra file ...finished in 1:44:56.298100 --- Sorting results --- ...finished in 0:00:09.558102 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 Total runtime: 3:18:55.473481 ```
fjahr commented 3 months ago

Nice, @Sjors and @brunoerg are a match ❤️ 😉

I am probably not going to match with anyone because I am missing a hand full of rpki repositories. I am not sure how that has happened but I have noticed previously the number of fetched repositories is not always consistent. I have opened an issue in kartograf for this and asked about this at rpki-client: https://github.com/fjahr/kartograf/issues/19

fjahr commented 3 months ago

To give some more details on how I detected this: When looking at my logs you can see Parsing 163900 ROAs and Result entries written: 401562, these are suspiciously low number, usually they are >200k and >500k respectively looking at my past runs. So I checked my rpki data folder and saw that I was missing several repository folders compared to my trial run yesterday.

$ comm -3 <(ls -1 ./projects/python/kartograf/data/1724193993/rpki/cache/ | sort) <(ls -1 ./projects/python/kartograf/data/1724248800/rpki/cache/ | sort)
ca.nat.moe/
rpki.arin.net/
rpki.miralium.net/
rpki.sailx.co/
rpki.tools.westconnect.ca/
dunxen commented 3 months ago

My final hash: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 (which matches @Sjors and @brunoerg).

P.S.: I see I only differed by having one less ROA than both Sjors and brunoerg, but got the same mapping :tada:

Logs ``` ./run map -w=1724248800 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.7 Using rpki-client version 9.1 (recommended). Coordinated launch mode: Waiting until 1724248800 (2024-08-21 16:00:00 SAST) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1724248800 (2024-08-21 14:00:00 UTC, local: 2024-08-21 16:00:00 SAST) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/dunxen/repos/fjahr/kartograf/data/1724248800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /home/dunxen/repos/fjahr/kartograf/data/1724248800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /home/dunxen/repos/fjahr/kartograf/data/1724248800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /home/dunxen/repos/fjahr/kartograf/data/1724248800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /home/dunxen/repos/fjahr/kartograf/data/1724248800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 33b58a819265c3a328b959fd2c8080431e1d486ff15e5e230e62343bd7de52c5 ...finished in 0:04:44.489998 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 4e2636c48738f81a244864c39f1afaead841305b10313d2cab4415bd145b918d Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: dcf3a00a94cdb4db6b8344f1d4e1e53215b2d5ddc320b1f8c84791cd73f63d6d Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 61c92d5cc0118b5928fb5eaf47e2c27f1c6d9ef49653be6045ab8ec85019ecb3 Downloading arin.db.gz Downloaded arin.db.gz, file hash: cadb285919d60b1652bf1048c25564163991c584bfcfba2c20524b0271b2df9c Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: 9cf49c26a48c1688d8674abeda8cc94c594d9fa180e8333ec46f562384b5299c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 714350fd59c119da833c5a0da515cee0f93de2b476095e9bae4993b9f85096bb Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 3cf7f061439d9932f46ce1a5c9fbae6d347f4736593fcf1a5fa7dbe5494ab8ee ...finished in 0:01:56.680255 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/08/routeviews-rv2-20240819-1200.pfx2as.gz Downloaded /home/dunxen/repos/fjahr/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 1b8827c21718ff44552534ad720b4ae323926a0e89891adefc9de298427e848d Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/08/routeviews-rv6-20240820-1200.pfx2as.gz Downloaded /home/dunxen/repos/fjahr/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 13f16d712e4451aba3b70ab2292dbf711322112004f902724b5c95d19b24ba4c ...finished in 0:00:10.180849 --- Validating RPKI --- Validating RPKI ROAs 310252 raw RKPI ROA files found. 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 310252/310252 [09:23<00:00, 550.84it/s] 310252 RKPI ROAs validated and saved to /home/dunxen/repos/fjahr/kartograf/out/1724248800/rpki/rpki_raw.json, file hash: 48330b014fbff4990a4ded6bcb213a6e7b6a7670cc84862113b256d154c52d15 ...finished in 0:10:00.671272 --- Parsing RPKI --- Parsing 310252 ROAs Result entries written: 543063 Duplicates found: 66213 Invalids found: 49997 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:08.543125 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/afrinic.db Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/apnic.db.route Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/apnic.db.route6 Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/arin.db Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/lacnic.db Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/ripe.db.route Parsing /home/dunxen/repos/fjahr/kartograf/out/1724248800/irr/ripe.db.route6 Found valid, unique entries: 1787236 ...finished in 0:04:34.581851 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111703 / 111703 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 111702 / 111702 | Finished filtering! Originally 1787236 entries filtered down to 378736 Merging base file with filtered extra file ...finished in 1:56:31.856867 --- Parsing Routeviews pfx2as --- Unzipping /home/dunxen/repos/fjahr/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/dunxen/repos/fjahr/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/dunxen/repos/fjahr/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/dunxen/repos/fjahr/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/dunxen/repos/fjahr/kartograf/out/1724248800/collectors/pfx2asn.txt ...finished in 0:02:09.103533 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76739 / 76739 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 76738 / 76738 | Finished filtering! Originally 1227815 entries filtered down to 386230 Merging base file with filtered extra file ...finished in 2:25:16.094068 --- Sorting results --- ...finished in 0:00:12.858833 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 Total runtime: 4:46:46.582898 ```
freerko commented 3 months ago

I've got the same final hash: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153

Logs ./run map -w=1724248800 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.7 Using rpki-client version 9.1 (recommended). Coordinated launch mode: Waiting until 1724248800 (2024-08-21 16:00:00 CEST) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1724248800 (2024-08-21 14:00:00 UTC, local: 2024-08-21 16:00:00 CEST) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/macbook/kartograf/data/1724248800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /Users/macbook/kartograf/data/1724248800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /Users/macbook/kartograf/data/1724248800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /Users/macbook/kartograf/data/1724248800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /Users/macbook/kartograf/data/1724248800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 7eedcd884f8c4e445e2f71bfef2400ea5eb38e3204f488309396668181f1d388 ...finished in 0:05:51.101644 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 4e2636c48738f81a244864c39f1afaead841305b10313d2cab4415bd145b918d Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: dcf3a00a94cdb4db6b8344f1d4e1e53215b2d5ddc320b1f8c84791cd73f63d6d Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 61c92d5cc0118b5928fb5eaf47e2c27f1c6d9ef49653be6045ab8ec85019ecb3 Downloading arin.db.gz Downloaded arin.db.gz, file hash: cadb285919d60b1652bf1048c25564163991c584bfcfba2c20524b0271b2df9c Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: 9cf49c26a48c1688d8674abeda8cc94c594d9fa180e8333ec46f562384b5299c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 714350fd59c119da833c5a0da515cee0f93de2b476095e9bae4993b9f85096bb Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 3cf7f061439d9932f46ce1a5c9fbae6d347f4736593fcf1a5fa7dbe5494ab8ee ...finished in 0:00:35.268736 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/08/routeviews-rv2-20240819-1200.pfx2as.gz Downloaded /Users/macbook/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 1b8827c21718ff44552534ad720b4ae323926a0e89891adefc9de298427e848d Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/08/routeviews-rv6-20240820-1200.pfx2as.gz Downloaded /Users/macbook/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 13f16d712e4451aba3b70ab2292dbf711322112004f902724b5c95d19b24ba4c ...finished in 0:00:16.222489 --- Validating RPKI --- Validating RPKI ROAs 310253 raw RKPI ROA files found. 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 310253/310253 [46:39<00:00, 110.84it/s] 310253 RKPI ROAs validated and saved to /Users/macbook/kartograf/out/1724248800/rpki/rpki_raw.json, file hash: 7bea87efc082b68ff84ca6b415b05bb3901af70171f360e4adf6ef4b780768ca ...finished in 0:46:56.610106 --- Parsing RPKI --- Parsing 310253 ROAs Result entries written: 543064 Duplicates found: 66213 Invalids found: 49997 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:51.670272 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/macbook/kartograf/out/1724248800/irr/lacnic.db Parsing /Users/macbook/kartograf/out/1724248800/irr/ripe.db.route Parsing /Users/macbook/kartograf/out/1724248800/irr/apnic.db.route6 Parsing /Users/macbook/kartograf/out/1724248800/irr/apnic.db.route Parsing /Users/macbook/kartograf/out/1724248800/irr/arin.db Parsing /Users/macbook/kartograf/out/1724248800/irr/afrinic.db Parsing /Users/macbook/kartograf/out/1724248800/irr/ripe.db.route6 Found valid, unique entries: 1787236 ...finished in 0:02:49.488553 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | Finished filtering! Originally 1787236 entries filtered down to 378735 Merging base file with filtered extra file ...finished in 2:41:04.478084 --- Parsing Routeviews pfx2as --- Unzipping /Users/macbook/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/macbook/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/macbook/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/macbook/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/macbook/kartograf/out/1724248800/collectors/pfx2asn.txt ...finished in 0:01:16.186861 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153476 / 153476 | Finished filtering! Originally 1227815 entries filtered down to 386230 Merging base file with filtered extra file ...finished in 2:50:08.021754 --- Sorting results --- ...finished in 0:00:07.936677 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 Total runtime: 6:29:58.257075
hebasto commented 3 months ago

I've got:

< snip >

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153
Total runtime: 10:27:29.130238
The log ``` $ ./run map -w=1724248800 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.6 Using rpki-client version 9.0. Please beware that running with the latest tested version (9.1) is recommend. Coordinated launch mode: Waiting until 1724248800 (2024-08-21 15:00:00 BST) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1724248800 (2024-08-21 14:00:00 UTC, local: 2024-08-21 15:00:00 BST) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/hebasto/git/kartograf/data/1724248800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /home/hebasto/git/kartograf/data/1724248800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /home/hebasto/git/kartograf/data/1724248800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /home/hebasto/git/kartograf/data/1724248800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /home/hebasto/git/kartograf/data/1724248800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 1a3a20cabde743b73396f10b45dd4cf04543bf76a8995f274be42049dcc0aebb ...finished in 0:06:40.367120 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 4e2636c48738f81a244864c39f1afaead841305b10313d2cab4415bd145b918d Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: dcf3a00a94cdb4db6b8344f1d4e1e53215b2d5ddc320b1f8c84791cd73f63d6d Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 61c92d5cc0118b5928fb5eaf47e2c27f1c6d9ef49653be6045ab8ec85019ecb3 Downloading arin.db.gz Downloaded arin.db.gz, file hash: cadb285919d60b1652bf1048c25564163991c584bfcfba2c20524b0271b2df9c Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: 9cf49c26a48c1688d8674abeda8cc94c594d9fa180e8333ec46f562384b5299c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 714350fd59c119da833c5a0da515cee0f93de2b476095e9bae4993b9f85096bb Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 3cf7f061439d9932f46ce1a5c9fbae6d347f4736593fcf1a5fa7dbe5494ab8ee ...finished in 0:01:13.436463 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/08/routeviews-rv2-20240819-1200.pfx2as.gz Downloaded /home/hebasto/git/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 1b8827c21718ff44552534ad720b4ae323926a0e89891adefc9de298427e848d Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/08/routeviews-rv6-20240820-1200.pfx2as.gz Downloaded /home/hebasto/git/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 13f16d712e4451aba3b70ab2292dbf711322112004f902724b5c95d19b24ba4c ...finished in 0:00:09.760872 --- Validating RPKI --- Validating RPKI ROAs 310252 raw RKPI ROA files found. 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 310252/310252 [20:43<00:00, 249.46it/s] 310252 RKPI ROAs validated and saved to /home/hebasto/git/kartograf/out/1724248800/rpki/rpki_raw.json, file hash: 3e208bc7f077dd7d6e95aa1837d3a0d1467ee4ffc80e898808e4fbc783c35791 ...finished in 0:21:02.704064 --- Parsing RPKI --- Parsing 310252 ROAs Result entries written: 543063 Duplicates found: 66213 Invalids found: 49997 Incompletes: 0 Non-ROA files: 0 ...finished in 0:02:14.107265 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/hebasto/git/kartograf/out/1724248800/irr/ripe.db.route Parsing /home/hebasto/git/kartograf/out/1724248800/irr/afrinic.db Parsing /home/hebasto/git/kartograf/out/1724248800/irr/apnic.db.route Parsing /home/hebasto/git/kartograf/out/1724248800/irr/lacnic.db Parsing /home/hebasto/git/kartograf/out/1724248800/irr/apnic.db.route6 Parsing /home/hebasto/git/kartograf/out/1724248800/irr/arin.db Parsing /home/hebasto/git/kartograf/out/1724248800/irr/ripe.db.route6 Found valid, unique entries: 1787236 ...finished in 0:07:34.531476 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223405 / 223405 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 223404 / 223404 | Finished filtering! Originally 1787236 entries filtered down to 378736 Merging base file with filtered extra file ...finished in 4:16:36.371818 --- Parsing Routeviews pfx2as --- Unzipping /home/hebasto/git/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/hebasto/git/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/hebasto/git/kartograf/data/1724248800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/hebasto/git/kartograf/out/1724248800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/hebasto/git/kartograf/out/1724248800/collectors/pfx2asn.txt ...finished in 0:02:08.809654 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153477 / 153477 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 153476 / 153476 | Finished filtering! Originally 1227815 entries filtered down to 386230 Merging base file with filtered extra file ...finished in 5:29:33.390867 --- Sorting results --- ...finished in 0:00:13.179403 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 Total runtime: 10:27:29.130238 ```
0xB10C commented 3 months ago 
--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153
Total runtime: 6:38:29.027651
fjahr commented 3 months ago

Hi all, thank you for participating is such high numbers. This is awesome! 🥳

Even though there might be still some more results coming in (please still post them if you haven't yet) it looks like 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153 is clearly a winner. I don't think we have ever had that many matches in one run, a good indication that keeping everyone on the same rpki-client version may be the right way to go for the future.

Can someone who got the matching hash please post their final_result.txt here? Thank you!

dunxen commented 3 months ago

Can someone who got the matching hash please post their final_result.txt here? Thank you!

It's too big to attach directly to a comment here (32 MB and limit is 25 MB), but here is a temporary link: [removed].

EDIT: See comment below (https://github.com/fjahr/asmap-data/issues/15#issuecomment-2304588468)

fjahr commented 3 months ago

Thanks @dunxen! It works if you zip it, putting it here so you don't have to maintain that link.

1724248800-final_result.txt.gz

fjahr commented 3 months ago

Closing this with great success 🥇

@jurraca has opened the PR to add the encoded version to the repo here: https://github.com/fjahr/asmap-data/pull/16 Would be great to get 1-2 more ACKs for verification there!

Thanks again and see you next time 🙂

Emzy commented 3 months ago

Run with rpki-client-portable 8.8:

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 0236b4f8c6e023e3bd201996c49f9f10852be0ca7e5952a4ba26fc1c00b71153
Total runtime: 19:28:46.398949