ooni / explorer

OONI Explorer: uncover evidence of internet censorship worldwide
https://explorer.ooni.org
BSD 3-Clause "New" or "Revised" License
71 stars 38 forks source link

Add support for getting data OONI Data Kraken API #892

Open hellais opened 10 months ago

hellais commented 10 months ago

The current target is only the MAT.

We support an environment flag for getting the data from the aggregation API implement in ooni/data, see: https://github.com/ooni/data/pull/45.

Here is a working example on how to run this:

  1. Edit .env.local
  2. Place the following in .env.local
    
    # Used only in local environments
    # Refer: https://nextjs.org/docs/basic-features/environment-variables

NEXT_PUBLIC_OONI_API=https://ams-pg-test.ooni.org NEXT_PUBLIC_OONI_DATA_API=https://data.ooni.org NEXT_PUBLIC_USER_FEEDBACK_API=https://ams-pg-test.ooni.org NEXT_PUBLIC_EXPLORER_URL=http://localhost:3100

RUN_GIT_COMMIT_SHA_SHORT=yarn --silent git:getCommitSHA:short RUN_GIT_COMMIT_SHA=yarn --silent git:getCommitSHA RUN_GIT_COMMIT_REF=yarn --silent git:getCommitRef RUN_GIT_COMMIT_TAGS=yarn --silent git:getReleasesAndTags


3. Run `yarn run dev`
4. Visit http://localhost:3100/chart/mat?probe_cc=ID&since=2023-10-23&until=2023-11-23&time_grain=day&axis_x=measurement_start_day&axis_y=domain&test_name=web_connectivity

Example live preview link: https://explorer-3ljbnbf5b-ooni1.vercel.app/chart/mat?probe_cc=ID&since=2023-10-23&until=2023-11-23&time_grain=day&axis_x=measurement_start_day&axis_y=domain&test_name=web_connectivity

## Notes about the implementation

For backward compatibility we implement an aggregation API which is compliant with the current API. The only difference is that it also supports a new field called `anomaly_threshold` which is set by default to `0.7` and used to determine if a particular measurement is to be considered anomalous or not based on the `loni` values.

It does however already support returning extra data which could eventually be used in an alternative future MAT view.

The schema of the returned data format can be inspected here: https://github.com/ooni/data/blob/5dd30c472da7a6eaa19071c5a49a510693e03f19/oonidata/fastapi/routers/aggregation.py#L43.

The interesting stuff is inside of the `loni_*` keys.

These are aggregate loni based likelyhood of blocking estimates calculated using the new analysis engine and are values that go from 0-1, where 1 indicates our belief in that particular blocking method being implemented.

Other new interesting keys are:
* `observation_count`: the unique count of observations used to determine a particular aggregate loni value
* `vantage_point_count `: a guess of the number of unique vantage points that contributed to this experiment result (estimated via `COUNT(DISTINCT location_network_type, location_network_asn, location_network_cc, location_resolver_asn)`)

Please note that the experiment results tables only contain data from the last few months and contrary to observation tables is not being updated regularly.

πŸ†•  you can now view the LoNI values by clicking on a bar in the MAT
vercel[bot] commented 10 months ago

The latest updates on your projects. Learn more about Vercel for Git β†—οΈŽ

Name Status Preview Comments Updated (UTC)
explorer βœ… Ready (Inspect) Visit Preview πŸ’¬ Add feedback Nov 22, 2023 2:15pm