Add support for generating observations for unsupported nettests - Githubissues

ooni / data

OONI Data CLI and Pipeline v5

https://docs.ooni.org/data

8 stars 4 forks source link

Add support for generating observations for unsupported nettests #32

Open hellais opened 1 year ago

hellais commented 1 year ago

Below I share the list of currently unsupported nettests. I group them into the ones for which we already have a deployed observation datamodel and those which might require a new observation table.

Works with existing observation tables:

[x] web_connectivity
[ ] urlgetter
[x] browser_web
[x] facebook_messenger
[x] telegram
[x] whatsapp
[x] signal
[x] stunreachability
[x] tor
[x] http_invalid_request_line
[x] http_header_field_manipulation
[ ] dnsping

Might require new observation tables:

[ ] riseupvpn
[ ] dash
[ ] ndt
[ ] openvpn
[ ] psiphon
[ ] torsf
[ ] vanilla_tor
[ ] dnscheck

You need to first implement a Model for the nettest parser: https://github.com/ooni/data/tree/main/oonidata/models and then a transform which converts it into relevant observation rows: https://github.com/ooni/data/tree/main/oonidata/transforms.

Use the make_* functions to convert the standard measurement keys into observations, for example see: https://github.com/ooni/data/blob/main/oonidata/transforms/nettests/telegram.py.