This issue is about improving the pipeline to correctly flag websites that are down. More specifically, this issue is about making sure the ETL pipeline recognizes all the states that are already recognized by OONI Explorer's parsing code.
To illustrate this issue, let us focus on a specific case.
Because of how the ETL pipeline handles this measurement, the search in Explorer has no way to indicate that the measurement actually tells us that the website is down. When we see the individual measurement, instead, Explorer correctly recognizes this case because it has a more comprehensive parsing of the measurement results.
This issue is about improving the pipeline to correctly flag websites that are down. More specifically, this issue is about making sure the ETL pipeline recognizes all the states that are already recognized by OONI Explorer's parsing code.
To illustrate this issue, let us focus on a specific case.
Since https://github.com/ooni/probe-cli/pull/953, webconnectivity LTE correctly detects cases where neither the probe nor the test helper resolved any IP address. We flag those cases as "website down". See, for example, this measurement: https://explorer.ooni.org/measurement/20220912T050804Z_webconnectivity_IT_30722_n1_DWMjAQ9rHm7ho2HT?input=http%3A%2F%2Fwww.hrcr.org%2F.
The ETL pipeline scores the same measurement as follows:
(See https://api.ooni.io/api/v1/measurement_meta?report_id=20220912T050804Z_webconnectivity_IT_30722_n1_DWMjAQ9rHm7ho2HT&input=http%3A%2F%2Fwww.hrcr.org%2F)
Because of how the ETL pipeline handles this measurement, the search in Explorer has no way to indicate that the measurement actually tells us that the website is down. When we see the individual measurement, instead, Explorer correctly recognizes this case because it has a more comprehensive parsing of the measurement results.