Collaborative Launch: 1730210400 (Tuesday, October 29th, 14:00 GMT)

[jurraca]

Next collaborative launch is planned for 1730210400.

./run map -w=1730210400 -irr -rv

The latest kartograf release is here. The requirements are Python 3.11 and rpki-client 9.3. We recommend you use Nix and run nix develop to enter an environment with the most recent dependency versions, including rpki-client 9.3.

This release will also generate a debug.log file at out/1730210400/debug.log to log rpki-client run info. This will be useful to understand discrepancies or RPKI failures if they occur.

[jurraca]

Happening two hours from now!

You'll see a warning that the RPKI version hasn't been tested passed 9.1 -- you can safely ignore and run the 9.3 version. Will get that fixed shortly.

[dunxen]

My result:

The SHA-256 hash of the result file is: 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce

Logs

``` ./run map -w=1730210400 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.8 Using rpki-client version 9.3 (recommended). Coordinated launch mode: Waiting until 1730210400 (2024-10-29 16:00:00 SAST) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1730210400 (2024-10-29 14:00:00 UTC, local: 2024-10-29 16:00:00 SAST) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 6cd35224cff561814f67b66a336b1a1c0d96d5a84882565f8da1a39cf72a36e8 ...finished in 0:02:47.040959 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 06b38332b7a943102f28183a9f9d228eecc979f610eb0d584d42328f2c8b3506 Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: c89fc97d30042e09bf52076702c1ce168246f47419859126955b2b3aec600f19 Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: e29b02a507a94c6fa44dbee7e6d56ebe04799c18f4972e7deecd28ea62ba124f Downloading arin.db.gz Downloaded arin.db.gz, file hash: 08bb92d3b3cb6ca927674e149f96299d2af42a9068aebf6e8f879dee5fc4fa4d Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: fc4146b26dddc5f4a2658fad86e75f663efb2909a191f58f91384a4ec9b4c90c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 7e51fb885feba90181387eb71e3f80e377a54dd63ac07d8d1378b0342383d258 Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 13e2665d306358db6f53d3ea0f483a83f371d027803a01a3af7c7c99351f7c48 ...finished in 0:01:00.416695 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/10/routeviews-rv2-20241027-1200.pfx2as.gz Downloaded /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: de48c5f0e488be2f32dbf3af8db23f168790b3cf3cdb90dd8ca6ac3e72ba4b7a Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/10/routeviews-rv6-20241028-1200.pfx2as.gz Downloaded /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: c1d03e9d306d66d6a1c3fae374f7077b6f94e0fa09483452a6c0abe6fa031189 ...finished in 0:00:12.936680 --- Validating RPKI --- Validating RPKI ROAs 326288 raw RKPI ROA files found. 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 326288/326288 [08:33<00:00, 635.36it/s] 326288 RKPI ROAs validated and saved to /home/dunxen/repos/fjahr/kartograf/out/1730210400/rpki/rpki_raw.json, file hash: 18472ec6170db5e05dbaa407ae889398a20d629c1190b2f77eef0ce68bdafbf9 ...finished in 0:09:08.937212 --- Parsing RPKI --- Parsing 326288 ROAs Result entries written: 557068 Duplicates found: 70434 Invalids found: 57827 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:10.319102 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/afrinic.db Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/apnic.db.route Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/apnic.db.route6 Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/arin.db Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/lacnic.db Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/ripe.db.route Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/ripe.db.route6 Found valid, unique entries: 1813212 ...finished in 0:04:35.926408 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113326 / 113326 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113325 / 113325 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113325 / 113325 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113325 / 113325 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 113325 / 113325 | Finished filtering! Originally 1813212 entries filtered down to 377073 Merging base file with filtered extra file ...finished in 2:01:31.230769 --- Parsing Routeviews pfx2as --- Unzipping /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/dunxen/repos/fjahr/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/dunxen/repos/fjahr/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/dunxen/repos/fjahr/kartograf/out/1730210400/collectors/pfx2asn.txt ...finished in 0:02:09.012038 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77568 / 77568 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 77567 / 77567 | Finished filtering! Originally 1241081 entries filtered down to 384523 Merging base file with filtered extra file ...finished in 2:32:12.673296 --- Sorting results --- ...finished in 0:00:12.763655 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce Total runtime: 4:55:02.716502 ```

[fjahr]

I have 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce.

(You can ignore the time tracking in my run, I did the download part in one location and then finished the processing in another.

Logs

``` $ ./run map -w=1730210400 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.8 Using rpki-client version 9.3 (recommended). Coordinated launch mode: Waiting until 1730210400 (2024-10-29 15:00:00 CET) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1730210400 (2024-10-29 14:00:00 UTC, local: 2024-10-29 15:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 457eb54bc20072ec302f5fa5b7f61bba11c6842f7f3ec6d8c28b8eabeab22f8e ...finished in 0:07:30.840128 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 06b38332b7a943102f28183a9f9d228eecc979f610eb0d584d42328f2c8b3506 Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: c89fc97d30042e09bf52076702c1ce168246f47419859126955b2b3aec600f19 Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: e29b02a507a94c6fa44dbee7e6d56ebe04799c18f4972e7deecd28ea62ba124f Downloading arin.db.gz Downloaded arin.db.gz, file hash: 08bb92d3b3cb6ca927674e149f96299d2af42a9068aebf6e8f879dee5fc4fa4d Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: fc4146b26dddc5f4a2658fad86e75f663efb2909a191f58f91384a4ec9b4c90c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 7e51fb885feba90181387eb71e3f80e377a54dd63ac07d8d1378b0342383d258 Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 13e2665d306358db6f53d3ea0f483a83f371d027803a01a3af7c7c99351f7c48 ...finished in 0:01:08.242644 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/10/routeviews-rv2-20241027-1200.pfx2as.gz Downloaded /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: de48c5f0e488be2f32dbf3af8db23f168790b3cf3cdb90dd8ca6ac3e72ba4b7a Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/10/routeviews-rv6-20241028-1200.pfx2as.gz Downloaded /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: c1d03e9d306d66d6a1c3fae374f7077b6f94e0fa09483452a6c0abe6fa031189 ...finished in 0:00:07.301552 --- Validating RPKI --- Validating RPKI ROAs 326295 raw RKPI ROA files found. 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 326295/326295 [3:23:04<00:00, 26.78it/s] 326295 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1730210400/rpki/rpki_raw.json, file hash: 190fa2cc1b5bdf3960c0ca450a76cdf39bc7d4a1def9932d0875cb73f617a0e7 ...finished in 0:39:54.935860 --- Parsing RPKI --- Parsing 326295 ROAs Result entries written: 557068 Duplicates found: 70434 Invalids found: 57834 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:05.939468 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/lacnic.db Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/ripe.db.route Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/apnic.db.route6 Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/apnic.db.route Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/arin.db Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/afrinic.db Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/ripe.db.route6 Found valid, unique entries: 1813212 ...finished in 0:03:55.230642 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 181322 / 181322 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181322 / 181322 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 181321 / 181321 | Finished filtering! Originally 1813212 entries filtered down to 377073 Merging base file with filtered extra file ...finished in 1:29:00.817305 --- Parsing Routeviews pfx2as --- Unzipping /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/FJ/projects/python/kartograf/out/1730210400/collectors/pfx2asn.txt ...finished in 0:02:01.592040 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 124109 / 124109 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 124108 / 124108 | Finished filtering! Originally 1241081 entries filtered down to 384523 Merging base file with filtered extra file ...finished in 1:40:25.020590 --- Sorting results --- ...finished in 0:00:16.142230 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce Total runtime: 6:49:01.317201 ```

[jurraca]

I got 37c70f403b65f9d7df887f73f0417fe401e8d0489b9c807f18351d190259d219

I seem to have more IRR and RouteViews data to parse than both of you.

Logs

``` --- Start Kartograf --- Kartograf version: 0.4.8 Using rpki-client version 9.3 (recommended). Coordinated launch mode: Waiting until 1730210400 (2024-10-29 14:00:00 WET) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1730210400 (2024-10-29 14:00:00 UTC, local: 2024-10-29 14:00:00 WET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/base/code/kartograf/data/1730210400/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /home/base/code/kartograf/data/1730210400/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /home/base/code/kartograf/data/1730210400/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /home/base/code/kartograf/data/1730210400/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /home/base/code/kartograf/data/1730210400/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: aa4e25b4207e7e9dc63c3bdc52ee21e8ddedf609adb03bf418ef8fffc2819b83 ...finished in 0:06:15.300586 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: 06b38332b7a943102f28183a9f9d228eecc979f610eb0d584d42328f2c8b3506 Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: c89fc97d30042e09bf52076702c1ce168246f47419859126955b2b3aec600f19 Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: e29b02a507a94c6fa44dbee7e6d56ebe04799c18f4972e7deecd28ea62ba124f Downloading arin.db.gz Downloaded arin.db.gz, file hash: 08bb92d3b3cb6ca927674e149f96299d2af42a9068aebf6e8f879dee5fc4fa4d Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: fc4146b26dddc5f4a2658fad86e75f663efb2909a191f58f91384a4ec9b4c90c Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: 7e51fb885feba90181387eb71e3f80e377a54dd63ac07d8d1378b0342383d258 Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 13e2665d306358db6f53d3ea0f483a83f371d027803a01a3af7c7c99351f7c48 ...finished in 0:04:22.929134 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/10/routeviews-rv2-20241027-1200.pfx2as.gz Downloaded /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: de48c5f0e488be2f32dbf3af8db23f168790b3cf3cdb90dd8ca6ac3e72ba4b7a Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/10/routeviews-rv6-20241028-1200.pfx2as.gz Downloaded /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: c1d03e9d306d66d6a1c3fae374f7077b6f94e0fa09483452a6c0abe6fa031189 ...finished in 0:00:07.072789 --- Validating RPKI --- Validating RPKI ROAs 326282 raw RKPI ROA files found. 100%|██████████████████████████████████████████████████████████████| 326282/326282 [23:13<00:00, 234.22it/s] 326282 RKPI ROAs validated and saved to /home/base/code/kartograf/out/1730210400/rpki/rpki_raw.json, file hash: 8a5cdcf52aca0d9ba07c56d482e1805ceec9cb32a4efa54e7d3ff46f2ef04e70 ...finished in 0:23:32.685831 --- Parsing RPKI --- Parsing 326282 ROAs Result entries written: 412435 Duplicates found: 41473 Invalids found: 210561 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:19.068106 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/base/code/kartograf/out/1730210400/irr/afrinic.db Parsing /home/base/code/kartograf/out/1730210400/irr/arin.db Parsing /home/base/code/kartograf/out/1730210400/irr/ripe.db.route6 Parsing /home/base/code/kartograf/out/1730210400/irr/apnic.db.route6 Parsing /home/base/code/kartograf/out/1730210400/irr/lacnic.db Parsing /home/base/code/kartograf/out/1730210400/irr/ripe.db.route Parsing /home/base/code/kartograf/out/1730210400/irr/apnic.db.route Found valid, unique entries: 1813212 ...finished in 0:06:55.376568 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 226652 / 226652 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226652 / 226652 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226652 / 226652 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226652 / 226652 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226651 / 226651 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226651 / 226651 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226651 / 226651 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 226651 / 226651 | Finished filtering! Originally 1813212 entries filtered down to 412660 Merging base file with filtered extra file ...finished in 3:08:55.204572 --- Parsing Routeviews pfx2as --- Unzipping /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/base/code/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/base/code/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/base/code/kartograf/out/1730210400/collectors/pfx2asn.txt ...finished in 0:03:23.099779 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 155135 / 155135 | Finished filtering! Originally 1241081 entries filtered down to 472611 Merging base file with filtered extra file ...finished in 4:36:25.700276 --- Sorting results --- ...finished in 0:00:19.232583 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 37c70f403b65f9d7df887f73f0417fe401e8d0489b9c807f18351d190259d219 Total runtime: 8:31:37.862229 ```

[jurraca]

OK, a bit low on participants, so thank you @dunxen for showing up! Could I ask you to upload your zipped final_result.txt? Would like to figure out where I differed.

Working on performance improvements which should make this script significantly faster.

[dunxen]

Could I ask you to upload your zipped final_result.txt? Would like to figure out where I differed.

Sure thing!

Here's the compressed result: 1730210400-final_result.txt.gz

[0xB10C]

I got 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce.

[jurraca]

Nice, looks like 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce is the "winner".

Looking at this a bit more closely, the difference for me seems to be RPKI validation results. We all got about the same IRR data, the same RPKI files and almost the same number of entries, but I have a lot more invalids, which seems to account for the difference in the result files. I see a ton of CRL not yet valid from several RIRs in debug.log. Looking into it.

[Emzy]

I got:

8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce

[jurraca]

Thanks everyone for participating!

Using the output file with hash 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce, I've opened issue https://github.com/asmap/asmap-data/pull/19 with the compressed file which you can verify. Would be good to get at least two ACKs before we make it the latest_asmap we point others to.