hellais
commented
5 years ago This is something which is also very important for OONI Explorer.
The current status is that in a one-off fashion data from the MLab tests were ingested into the metadb, but they are going to soon start becoming stale and old.
If you look at the #mlab channel backlog you can see what has been done so far, but I will try to recap some of it here.
The easiest way to ingest MLab measurement data is by using the bigquery tables and running queries like: https://github.com/m-lab/mlab-vis-pipeline/tree/master/dataflow/data/bigtable/queries.
In particular the queries we are mostly interested in are something along the lines of:
#legacySQL
SELECT client_asn_number,
client_asn_name,
client_country_code,
SUM(rtt_sum) / SUM(rtt_count) AS rtt_avg,
AVG(packet_retransmit_rate) AS retransmit_avg,
nth(51, quantiles(download_speed_mbps, 101)) AS download_speed_mbps_median,
nth(51, quantiles(upload_speed_mbps, 101)) AS upload_speed_mbps_median,
COUNT(*) AS count
FROM [mlab-oti.data_viz.all_ip_by_hour]
WHERE LENGTH(client_asn_name) > 0
AND LENGTH(client_asn_number) > 0
AND local_test_date >= '2018-03-01 00:00:00'
AND local_test_date < '2018-04-01 00:00:00'
GROUP BY client_asn_number,
client_asn_name,
client_country_code
ORDER BY client_country_code;I did this manually at some point and populated the table of the MetaDB called ooexpl_netinfo.
What needs to happen is setup some form of system that automatically runs this sync periodically so that the data stays fresh.
I have a google account which has query permission on the BigQuery table and confirmed with the folks of MLab that it is fine if we run the above query on their dataset every 24h.
@FedericoCeratto any thoughts on this and where we can put in this logic?
So it's exciting that NDT and DASH have been rolled out and integrated into OONI Probe mobile apps, and that we're collecting loads of network measurements from these tests on a daily basis.
But how will this data be analyzed? How will we abstract value from it? What is the data analysis methodology?
I'm opening this ticket to encourage brainstorming and discussions around the creation of a methodology to analyze the incoming NDT and DASH network measurement data.
We may be interested in creating a data analysis methodology geared towards (if possible):
Detecting/examining cases of throttling
Identifying potential net neutrality violations
If we are able to successfully develop a methodology for examining and detecting the above, we would be in a unique position to support internet freedom community members who are working on these issues.
cc' @bassosimone @darkk @hellais