search

ooni / explorer

OONI Explorer: uncover evidence of internet censorship worldwide
https://explorer.ooni.org
BSD 3-Clause "New" or "Revised" License
74 stars 38 forks source link

measurement page: add support for showing the network name #45

Open hellais opened 5 years ago

hellais commented 5 years ago

Currently we only show the ASN, but we don't show the network name as this is not being exposed via the OONI API or submitted as part of the measurement.

This ticket is about extending OONI Explorer to also resolve a ASN into a proper network name.

hellais commented 5 years ago

I did a bit of research into this, namely.

I downloaded the GeoLite2 ASN database from: https://dev.maxmind.com/geoip/geoip2/geolite2/ and wrote a simple script to parse it and check it for consistencies:

import csv
import json

PATH = "GeoLite2-ASN-CSV_20190108/GeoLite2-ASN-Blocks-IPv4.csv"
def main():
    asn_map = {}
    with open(PATH) as in_file:
        csv_reader = csv.reader(in_file)
        next(csv_reader)
        for row in csv_reader:
            network, asn, org = row
            if org == "":
                continue
            if asn in asn_map:
                if asn_map[asn] != org:
                    print("WARNING: {}: {} != {}".format(asn, asn_map[asn], org))
            asn_map[asn] = org
    with open('asn-map.json', 'w+') as out_file:
        json.dump(asn_map, out_file)

main()

The resulting asn-map.json is 3MB in size.

In doing this exercise I noticed that there are some inconsistencies in the maxmind database. Namely that certain ASNs are marked as having two different org names listed for them, depending on the netblock in question.

Here is a summary of them:

WARNING: 21986: AT&T Services, Inc. != Cogent Communications
WARNING: 393266: Level 3 Parent, LLC != Cogent Communications
WARNING: 54753: AT&T Services, Inc. != Level 3 Parent, LLC
WARNING: 27205: Level 3 Parent, LLC != MCI Communications Services, Inc. d/b/a Verizon Business
WARNING: 19111: AT&T Services, Inc. != MCI Communications Services, Inc. d/b/a Verizon Business
WARNING: 27010: AT&T Services, Inc. != Windstream Communications LLC
WARNING: 33628: AT&T Services, Inc. != Sprint
WARNING: 32282: AT&T Services, Inc. != Sprint
WARNING: 27205: MCI Communications Services, Inc. d/b/a Verizon Business != SunGard Availability Services LP
WARNING: 54753: Level 3 Parent, LLC != CenturyLink Communications, LLC
WARNING: 27426: AT&T Services, Inc. != Cox Communications Inc.
WARNING: 46228: Level 3 Parent, LLC != Internap Corporation
WARNING: 30018: Masergy Communications != Internap Corporation
WARNING: 30672: AT&T Services, Inc. != Level 3 Parent, LLC
WARNING: 26757: MCI Communications Services, Inc. d/b/a Verizon Business != Sprint
WARNING: 55040: Cogent Communications != Sprint
WARNING: 19658: AT&T Corp. != MCI Communications Services, Inc. d/b/a Verizon Business
WARNING: 40640: Cogent Communications != MCI Communications Services, Inc. d/b/a Verizon Business
WARNING: 30104: Cablevision Systems Corp. != Lightower Fiber Networks I, LLC
WARNING: 12122: AT&T Services, Inc. != AT&T Corp.
WARNING: 35948: MCI Communications Services, Inc. d/b/a Verizon Business != AT&T Corp.
WARNING: 26522: Cogent Communications != Net Access Corporation
WARNING: 62495: Level 3 Parent, LLC != CenturyLink Communications, LLC
WARNING: 30104: Lightower Fiber Networks I, LLC != Cablevision Systems Corp.
WARNING: 14350: Cogent Communications != Cablevision Systems Corp.
WARNING: 40494: AT&T Services, Inc. != Cox Communications Inc.
WARNING: 46579: AT&T Services, Inc. != Atlantic Metro Communications, LLC
WARNING: 26757: Sprint != Solera Holdings, Inc.
WARNING: 14360: Cogent Communications != Internap Corporation
WARNING: 26016: Lightower Fiber Networks I, LLC != Internap Corporation
WARNING: 12122: AT&T Corp. != AT&T Services, Inc.
WARNING: 27205: SunGard Availability Services LP != COLT Technology Services Group Limited
WARNING: 393377: AT&T Services, Inc. != Charter Communications Inc
WARNING: 36843: Florida State University != Florida Atlantic University
WARNING: 62616: Level 3 Parent, LLC != NTT America, Inc.
WARNING: 19111: MCI Communications Services, Inc. d/b/a Verizon Business != Lightower Fiber Networks I, LLC
WARNING: 36843: Florida Atlantic University != Florida State University
WARNING: 32422: AT&T Services, Inc. != Connecticut Hospital Assoc.
WARNING: 19658: MCI Communications Services, Inc. d/b/a Verizon Business != Fibernet Direct
WARNING: 36843: Florida State University != CenturyLink Communications, LLC
WARNING: 16808: Internap Corporation != California Education and Research Federation Network
WARNING: 65543: Spilsby Internet Solutions != Stephouse Networks
WARNING: 6517: GTT Communications Inc. != Reliance Globalcom Limited
WARNING: 46579: Atlantic Metro Communications, LLC != InfoRelay Online Systems, Inc.
WARNING: 33394: CenturyLink Communications, LLC != Northwest Nexus Inc.
WARNING: 20169: Nextera Communications LLC != Sprint
WARNING: 30254: MCI Communications Services, Inc. d/b/a Verizon Business != Level 3 Parent, LLC
WARNING: 20089: MCI Communications Services, Inc. d/b/a Verizon Business != CenturyLink Communications, LLC
WARNING: 32596: GTT Communications Inc. != CenturyLink Communications, LLC
WARNING: 22538: AT&T Services, Inc. != Level 3 Parent, LLC
WARNING: 27289: Comcast Cable Communications, LLC != CenturyLink Communications, LLC
WARNING: 6517: Reliance Globalcom Limited != GTT Communications Inc.
WARNING: 30254: Level 3 Parent, LLC != MCI Communications Services, Inc. d/b/a Verizon Business
WARNING: 393385: Comcast Cable Communications, LLC != AT&T Services, Inc.
WARNING: 40920: Masergy Communications != Zayo Bandwidth
WARNING: 33051: AT&T Corp. != Cox Communications Inc.
WARNING: 19330: Hurricane Electric LLC != IX Reach International Ltd
WARNING: 23474: Zayo Bandwidth != Blue Ridge Websoft, LLC
WARNING: 25786: Cogent Communications != Internap Corporation
WARNING: 30550: Level 3 Parent, LLC != AT&T Corp.
WARNING: 29855: Comcast Cable Communications, LLC != Level 3 Parent, LLC
WARNING: 27393: MCI Communications Services, Inc. d/b/a Verizon Business != GTT Communications Inc.
WARNING: 393320: Comcast Cable Communications, LLC != Zayo Bandwidth
WARNING: 46978: Internap Corporation != CenturyLink Communications, LLC
WARNING: 65552: IP-Only Networks AB != IFX Corporation

A lot of them seem to be related to Level 3, Cogent, AT&T, Comcast & CenturyLink.

I wonder what we should do for these cases, because by having resolved the IP to a AS number we don't actually know where what the original netblock was. We probably should pick one of these and it would probably be a good idea to understand why these inconsistencies exist to make a smarter choice.

cc @bassosimone @darkk

darkk commented 5 years ago

FYI, there are other sorts of inconsistencies as well. E.g. originas zone from routeviews can produce following records (two ASNs for exactly the same prefix):

  asn  |    origin     |                      asname
-------+---------------+---------------------------------------------------
 17916 | 20.134.0.0/20 | CSC-IGN-AUNZ-AP Computer Sciences Corporation, AU
  7474 | 20.134.0.0/20 | OPTUSCOM-AS01-AU SingTel Optus Pty Ltd, AU
hellais commented 5 years ago

I don't think we will manage to do this in the stable as it requires a fair amount of backend and pipeline work as well.

sarathms commented 5 years ago

In that case, I think we should change the layout of the relevant section and make it a 3 column layout on desktop like below. Significantly changes the first impression of the page. @holantonela @hellais

image

It is already one-column on mobile. image

holantonela-zz commented 5 years ago

I suggested this before, but @hellais wants to have room for the long asname. Could we stick to the three columns until we have the asname available @hellais?

hellais commented 4 years ago

This may be relevant to the work @FedericoCeratto is doing with the ASN database: https://github.com/ooni/probe-engine/issues/336

hellais commented 3 years ago

We might want to generate this using the asn-datatbase generator code here: https://github.com/ooni/asn-db-generator or potentially put it in a timer inside of the pipeline.

hellais commented 2 years ago

We now have this field inside of the probe generated measurements directly (it's the probe_network_name field). Perhaps we can start displaying it for measurements that include it and leave it blank in case we don't spot it.

Eventually the pipeline should backfill them.

hellais commented 2 years ago

We can just add it to the header and leave the AS in parenthesis like so: Screenshot 2022-08-04 at 16 12 19