replacement for MaxMind ASN mmdb

bassosimone commented 4 years ago

Part of https://github.com/ooni/probe-engine/issues/269

bassosimone commented 4 years ago

We looked into this topic with @FedericoCeratto and we think we should look into:

These pages also contains relevant information: https://www.team-cymru.com/IP-ASN-mapping.html and https://github.com/t2mune/mrtparse#mrtparse

bassosimone commented 4 years ago

We have learned that the best database is the RIPE one. We can extract data and we can most likely reuse the data freely. There is also an available history.

Different boxes have different points of view. We used the rrc04 box and we downloaded. We want to download latest-bview.gz. This file is the dump of the routing table. We used the following command:

mrt-print-all latest-bview.gz | head -n300

We obtain data like the following:

---------------------------------------------------------------
MRT Header
    Timestamp: 1581955200(2020-02-17 16:00:00)
    Type: 13(TABLE_DUMP_V2)
    Subtype: 2(RIB_IPV4_UNICAST)
    Length: 288
RIB_IPV4_UNICAST
    Sequence Number: 6
    Prefix Length: 24
    Prefix: 1.0.6.0
    Entry Count: 6
    Peer Index: 4
    Originated Time: 1581852431(2020-02-16 11:27:11)
    Attribute Length: 36
    Path Attribute Flags/Type/Length: 0x40/1/1
        ORIGIN: 0(IGP)
    Path Attribute Flags/Type/Length: 0x40/2/22
        AS_PATH
            Path Segment Type: 2(AS_SEQUENCE)
            Path Segment Length: 5
            Path Segment Value: 29222 6939 4826 38803 56203
[snip]

From there we can get the prefix, the prefix length. The ASN (56203) is part of the Path Segment Value, but we're not sure we can consider this as being true in general.

bassosimone commented 4 years ago

We can probably consider using this tool for writing the database:

https://github.com/maxmind/MaxMind-DB-Writer-perl.

This blog post is possibly relevant:

https://blog.maxmind.com/2015/09/29/building-your-own-mmdb-database-for-fun-and-profit/.

We don't know whether it's overkill to package as MaxMind.

Maybe using a CSV is enough?

hellais commented 4 years ago

I am going to suggest we close this issue in favour of https://github.com/ooni/probe-engine/issues/507 where we should add concrete next steps.

ooni / probe-engine

replacement for MaxMind ASN mmdb #336