Collaborate with Netalitica on test list related improvements

[agrabeli]

We will continue to collaborate with Netalitica by reviewing all test list updates by their researchers, sharing feedback, and helping out with getting test list updates merged on GitHub. We will also continue to update test lists on an ongoing basis, particularly as new censorship events emerge and as part of our research. We will continue to engage community members with test list research and we will update test lists with any new URLs we receive.

To encourage greater volunteer contributions and help remove barriers to participation, we will work on exploring how to simplify the process of contributing to test lists and how to otherwise support Netalitica’s test list work. This may involve re-structuring the data model of test lists, changing the tooling for automatically reviewing test list contributions, and otherwise writing software to solve problems related to updating test lists.

We will also mine OONI data to identify tested URLs that are not currently included in test lists (and add them). As OONI Probe users have the opportunity to test websites of their choice (independently from test lists) -- either by testing single URLs via the OONI Probe app or by using OONI Run -- many tested URLs are currently missing from test lists and are only tested a few times, rather than on an ongoing basis. As a result, we only have a limited volume of measurements for such URLs (which are potentially very interesting to test) and confirming their blocking can be more challenging.

Often, such URLs include those that community members test (using OONI Run) in response to emergent censorship events, or as part of a censorship measurement campaign leading up to and during a political event (such as elections or a planned demonstration). To ensure that those URLs are monitored over a longer period of time, we will mine OONI data to identify those URLs and we will collaborate with Netalitica on adding those URLs to their respective test lists (which may require the participation of local researchers to help with categorizing those URLs).

In summary, the OONI team will work closely with Netalitica to understand what needs and gaps are missing in OONI tools, as well as how it’s possible to further automate or streamline the work on test lists performed by Netalitica.

[agrabeli]

@hellais I have assigned you because this activity also involves mining OONI data to identify URLs that should be added to test lists.

[agrabeli]

Ticket for the creation of a script to automatically update URLs to HTTPS: https://github.com/ooni/ooni.org/issues/1226

[agrabeli]

Ticket for the creation of a script to automatically delete expired and parked domains: https://github.com/ooni/ooni.org/issues/1227 (but this might be done by Roskomsvoboda hackathon participants)

[agrabeli]

Ticket on mining OONI data to identify manually tested URLs: https://github.com/ooni/ooni.org/issues/1228

[hellais]

This has been done, see all the issues linked here.