search

asmap / asmap-data

Demo repository for how a similar repository could be used in Bitcoin Core
MIT License
6 stars 2 forks source link

Collaborative Launch 1710424800 (Thursday, March 14, 2024 2:00:00 PM GMT) #11

Closed fjahr closed 8 months ago

fjahr commented 8 months ago

Next collaborative launch is planned for 1710424800. Please make sure to use Kartograf v0.4.4.

./run map -w=1710424800 -irr -rv
fjahr commented 8 months ago

My result is 9c44adcb3e6b48d1df19eb415cc6ea8801d921b02337ed7c6f52b42780a32db5.

Full logs ``` $ ./run map -w=1710424800 -irr -rv --- Start Kartograf --- Kartograf version: 0.4.4 Using rpki-client version 8.6. Coordinated launch mode: Waiting until 1710424800 (2024-03-14 15:00:00 CET) to launch mapping process. Countdown: 0 second(s) Starting... The epoch for this run is: 1710424800 (2024-03-14 14:00:00 UTC, local: 2024-03-14 15:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1710424800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1710424800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1710424800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1710424800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1710424800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 346476991998a6932e2cbc960c788160bdabb01e4e5973eb40fbfda5510e86e7 ...finished in 0:04:52.740056 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: bb276d395eae951b22cfc85371e49fecd38f96852ab57c6bd0932efc30c60fb2 Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: b7eb77a2d0c0605abb96a6e7675134995f578fa729f8b3e7975b7ff8d1a94a2e Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 06007ca643aae254ffe54d14be27ebe9c94bd2ebabe97a46a3227373a9009a9e Downloading arin.db.gz Downloaded arin.db.gz, file hash: 6b8f3a982e6044ba71f23ef9fd03b4a03beb54ba1ec95b7526f443bb7e8ec8a5 Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: ad3438babbadb65267d55df26ad5e9c68ae1ec87b538736003a4605eebfae111 Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: da87e8fe6517a47712b9ac6f0e33441e8f8dfa977324e2b2b4762ad663cd8034 Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 5dd9da9a3da24bc730cc8bd969f0e19d44a39d0c5e5197122f8b1ec5ed3a8d00 ...finished in 0:00:58.819432 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/03/routeviews-rv2-20240312-0800.pfx2as.gz Downloaded /Users/FJ/projects/python/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 0a48282e8cb84758497b36300d0ac111edbe04e74f56f6a5b6d4577703dc6e16 Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/03/routeviews-rv6-20240313-1200.pfx2as.gz Downloaded /Users/FJ/projects/python/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 3d5c77a51658b87df80a362b443123cee60daec6e0e00d1d9b4e45ac16832584 ...finished in 0:00:06.882231 --- Validating RPKI --- Validating RPKI ROAs 233189 raw RKPI ROA files found. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 233189/233189 [34:59<00:00, 111.08it/s] 233189 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1710424800/rpki/rpki_raw.json, file hash: a30b49215ee7d07effaf40efec77151778bd8c6140331c909f20581151805055 ...finished in 0:35:16.691697 --- Parsing RPKI --- Parsing 233189 ROAs Result entries written: 473238 Duplicates found: 60036 Invalids found: 33867 Incompletes: 0 Non-ROA files: 0 ...finished in 0:00:56.832154 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/lacnic.db Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/ripe.db.route Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/apnic.db.route6 Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/apnic.db.route Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/arin.db Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/afrinic.db Parsing /Users/FJ/projects/python/kartograf/out/1710424800/irr/ripe.db.route6 Found valid, unique entries: 1697927 ...finished in 0:03:45.389865 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169793 / 169793 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169792 / 169792 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169792 / 169792 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 169792 / 169792 | Finished filtering! Originally 1697927 entries filtered down to 390237 Merging base file with filtered extra file ...finished in 1:12:11.944323 --- Parsing Routeviews pfx2as --- Unzipping /Users/FJ/projects/python/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1710424800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /Users/FJ/projects/python/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /Users/FJ/projects/python/kartograf/out/1710424800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /Users/FJ/projects/python/kartograf/out/1710424800/collectors/pfx2asn.txt ...finished in 0:02:01.263331 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120024 / 120024 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120023 / 120023 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 120023 / 120023 | Finished filtering! Originally 1200238 entries filtered down to 404568 Merging base file with filtered extra file ...finished in 1:35:19.417913 --- Sorting results --- ...finished in 0:00:11.911702 --- Finishing Kartograf --- The SHA-256 hash of the result file is: 9c44adcb3e6b48d1df19eb415cc6ea8801d921b02337ed7c6f52b42780a32db5 Total runtime: 3:35:43.275116 ```
brunoerg commented 8 months ago

My result: 

Finished filtering! Originally 1200238 entries filtered down to 480566
Merging base file with filtered extra file
...finished in 0:59:35.860123

--- Sorting results ---

...finished in 0:00:08.962830

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 7bf42f66dd2975f630b0502830c4ef632e313a803101eb12f99c2057fe64ff23
Total runtime: 2:31:06.090485

EDIT: Sorry, I was running kartograf 0.4.2. Feel free to ignore.

fjahr commented 8 months ago

@brunoerg can you post the full logs so we can see where the divergence is?

dunxen commented 8 months ago

Result: 8bd8c51b1c2479b2df06f16dda9c62abf21f7f77af4e00746206c0715ecd170c Total runtime: 4:01:58.991689

Looking for them, but think I may have been stupid enough to lose the full logs... I was definitely on 0.4.4, though.

0xB10C commented 8 months ago

The SHA-256 hash of the result file is: d55d566b0b1cc576e10ce13fe3ded6cdb3d6a17b887470a651c13b4884218fe4 Total runtime: 5:52:43.052293

Full log ``` --- Start Kartograf --- Kartograf version: 0.4.4 Using rpki-client version 8.6. Coordinated launch mode: Waiting until 1710424800 (2024-03-14 15:00:00 CET) to launch mapping process. Countdown: 1 day(s), 3 hour(s), 29 minute(s), 59 second(s) Countdown: 1 day(s), 3 hour(s), 29 minute(s), 58 second(s) Countdown: 1 day(s), 3 hour(s), 29 minute(s), 57 second(s) Countdown: 1 day(s), 3 hour(s), 29 minute(s), 56 second(s) CoCountdown: 0 second(s) 3 hour(s), 29 minute(s), 54 second(s) Starting... The epoch for this run is: 1710424800 (2024-03-14 14:00:00 UTC, local: 2024-03-14 15:00:00 CET) --- Fetching RPKI --- Downloaded TAL for AFRINIC to /home/b10c/kartograf/data/1710424800/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35 Downloaded TAL for APNIC to /home/b10c/kartograf/data/1710424800/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94 Downloaded TAL for ARIN to /home/b10c/kartograf/data/1710424800/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48 Downloaded TAL for LACNIC to /home/b10c/kartograf/data/1710424800/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a Downloaded TAL for RIPE to /home/b10c/kartograf/data/1710424800/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb Downloading RPKI Data, this may take a while. Downloaded RPKI Data, hash sum: 6ce52b80c269e708a0dcdaccd74f5233cf8e1edb1281199b67b4e6734c9996ad ...finished in 0:04:14.074906 --- Fetching IRR --- Downloading afrinic.db.gz Downloaded afrinic.db.gz, file hash: bb276d395eae951b22cfc85371e49fecd38f96852ab57c6bd0932efc30c60fb2 Downloading apnic.db.route.gz Downloaded apnic.db.route.gz, file hash: b7eb77a2d0c0605abb96a6e7675134995f578fa729f8b3e7975b7ff8d1a94a2e Downloading apnic.db.route6.gz Downloaded apnic.db.route6.gz, file hash: 06007ca643aae254ffe54d14be27ebe9c94bd2ebabe97a46a3227373a9009a9e Downloading arin.db.gz Downloaded arin.db.gz, file hash: 6b8f3a982e6044ba71f23ef9fd03b4a03beb54ba1ec95b7526f443bb7e8ec8a5 Downloading lacnic.db.gz Downloaded lacnic.db.gz, file hash: ad3438babbadb65267d55df26ad5e9c68ae1ec87b538736003a4605eebfae111 Downloading ripe.db.route.gz Downloaded ripe.db.route.gz, file hash: da87e8fe6517a47712b9ac6f0e33441e8f8dfa977324e2b2b4762ad663cd8034 Downloading ripe.db.route6.gz Downloaded ripe.db.route6.gz, file hash: 5dd9da9a3da24bc730cc8bd969f0e19d44a39d0c5e5197122f8b1ec5ed3a8d00 ...finished in 0:00:39.434000 --- Fetching Routeviews pfx2as --- Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/03/routeviews-rv2-20240312-0800.pfx2as.gz Downloaded /home/b10c/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: 0a48282e8cb84758497b36300d0ac111edbe04e74f56f6a5b6d4577703dc6e16 Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/03/routeviews-rv6-20240313-1200.pfx2as.gz Downloaded /home/b10c/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: 3d5c77a51658b87df80a362b443123cee60daec6e0e00d1d9b4e45ac16832584 ...finished in 0:00:06.739747 --- Validating RPKI --- Validating RPKI ROAs 233013 raw RKPI ROA files found. 100%|█████████████████████████████████████████████████████████████████████████████████| 233013/233013 [22:04<00:00, 175.89it/s] 233013 RKPI ROAs validated and saved to /home/b10c/kartograf/out/1710424800/rpki/rpki_raw.json, file hash: 034c59aa9518ea2fc7257b53bf54423e893f50487e8a7aeaa650cc230a1fac8d ...finished in 0:22:19.925576 --- Parsing RPKI --- Parsing 233013 ROAs Result entries written: 473095 Duplicates found: 60032 Invalids found: 33844 Incompletes: 0 Non-ROA files: 0 ...finished in 0:01:38.159807 --- Parsing IRR --- Extracting afrinic.db.gz Extracting apnic.db.route.gz Extracting apnic.db.route6.gz Extracting arin.db.gz Extracting lacnic.db.gz Extracting ripe.db.route.gz Extracting ripe.db.route6.gz Parsing /home/b10c/kartograf/out/1710424800/irr/afrinic.db Parsing /home/b10c/kartograf/out/1710424800/irr/apnic.db.route Parsing /home/b10c/kartograf/out/1710424800/irr/apnic.db.route6 Parsing /home/b10c/kartograf/out/1710424800/irr/arin.db Parsing /home/b10c/kartograf/out/1710424800/irr/lacnic.db Parsing /home/b10c/kartograf/out/1710424800/irr/ripe.db.route Parsing /home/b10c/kartograf/out/1710424800/irr/ripe.db.route6 Found valid, unique entries: 1697927 ...finished in 0:07:02.146329 --- Merging RPKI and IRR data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 282988 / 282988 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 282988 / 282988 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 282988 / 282988 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 282988 / 282988 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 282988 / 282988 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 282987 / 282987 | Finished filtering! Originally 1697927 entries filtered down to 390265 Merging base file with filtered extra file ...finished in 2:13:50.113095 --- Parsing Routeviews pfx2as --- Unzipping /home/b10c/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip4.txt.gz Formatting /home/b10c/kartograf/out/1710424800/collectors/routeviews_pfx2asn_ip4.txt Unzipping /home/b10c/kartograf/data/1710424800/collectors/routeviews_pfx2asn_ip6.txt.gz Formatting /home/b10c/kartograf/out/1710424800/collectors/routeviews_pfx2asn_ip6.txt Cleaning /home/b10c/kartograf/out/1710424800/collectors/pfx2asn.txt ...finished in 0:03:25.609384 --- Merging Routeviews and base data --- Parse base file to numpy arrays Parse extra file to Pandas DataFrame Filtering extra prefixes that were already included in the base file: 100.00% :::::::::::::::::::::::::::::::::::::::: | 200040 / 200040 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 200040 / 200040 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 200040 / 200040 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 200040 / 200040 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 200039 / 200039 | 100.00% :::::::::::::::::::::::::::::::::::::::: | 200039 / 200039 | Finished filtering! Originally 1200238 entries filtered down to 404578 Merging base file with filtered extra file ...finished in 2:59:05.275488 --- Sorting results --- ...finished in 0:00:19.663644 --- Finishing Kartograf --- The SHA-256 hash of the result file is: d55d566b0b1cc576e10ce13fe3ded6cdb3d6a17b887470a651c13b4884218fe4 Total runtime: 5:52:43.052293 ```

Seems like @fjahr and me have gotten different RPKI data in the download.


# b10c:
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 6ce52b80c269e708a0dcdaccd74f5233cf8e1edb1281199b67b4e6734c9996ad

# fjahr:
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 346476991998a6932e2cbc960c788160bdabb01e4e5973eb40fbfda5510e86e7
Emzy commented 8 months ago

The SHA-256 hash of the result file is: 9b54bf3ca12b70bfbc065a816680c4b536580a3508483cf3b1df4b0d41f1421d Total runtime: 16:50:51.287300

I lost my log, because of resizing the terminal while running.

fjahr commented 8 months ago

@Emzy @brunoerg @dunxen @0xB10C

Thanks everyone, looks like it's the first time we have no match at all 😞 I would like to do another one next week on Monday, same time, and I hope we are more lucky that time.

The new issue is here: https://github.com/fjahr/asmap-data/issues/12

I hope we can get a match this time :)