Remove 'negative' statistics from the statistics page

[ajparsons]

Looking at the WDTK statistics page.

I've been writing up our current position on the automated statistics on the other services, and would like to suggest removing the following to bring into line with FixMyStreet, which now only lists top authorities rather than a ranking:

• Remove 'Public bodies with the fewest successful requests'
• Remove 'Public bodies with most overdue requests'

Reasoning for this:

• New general approach that statistics that need disclaimers are probably not statistics we would stand by if someone asked. Currently the disclaimers on this list include 'a known problem that will be fixed in a later release', and I think that one is significant enough I'm not comfortable in negative judgements.
• Issue of user generated data consistent across FixMyStreet and WhatDoTheyKnow - depend on users telling us things, and they do not do so consistently. The most useful comparisons between authorities (batch requests) are the ones with the least data because it is so much work for requesters to update.
• These statistics need some love, especially to understand change over time (which was less significant when the screen was created, and 2020 will cause some disruption). They're not going to get this in the near term.
• Difference between kind of requests on WDTK and elsewhere questions general meaning of these statistics.
• I'm a little suspicious 'most overdue' is reflecting technical problems as the same authorities in other cases trigger automated replies.
• Research agenda has been aimed at making more comparisons between WDTK data and real world numbers. This should continue, with an eye for how these automated measures can be improved.
• Given reality of low priority of improvement, should reduce 'negative' requests in the short term rather than waiting for an overhaul.

Any opinions?

[garethrees]

These statistics need some love… They're not going to get this in the near term.

The code generating these stats is quite a mess so I think this assessment is correct.

Issue of user generated data consistent across FixMyStreet and WhatDoTheyKnow - depend on users telling us things, and they do not do so consistently. The most useful comparisons between authorities (batch requests) are the ones with the least data because it is so much work for requesters to update.

I'd be happier to retain these if we excluded requests awaiting classification from the counts, so that it's a much better reflection of "where we've been actively told what the result was, here are the overall stats". There's still some margin for error given user-contributed data, but it would feel more justifiable when we're excluding various automated or inferred classifications.

Difference between kind of requests on WDTK and elsewhere questions general meaning of these statistics. … I'm a little suspicious 'most overdue' is reflecting technical problems as the same authorities in other cases trigger automated replies.

Don't really understand what you mean here, but I think there is a good case that WDTK can give an indication of which authorities are fulfilling their obligations. I think "delivery errors" in particular highlight roadblocks to FOI. They're forever changing their addresses to add extra burden for the citizen. IMO all authorities subject to the act should have foi@THEIR_DOMAIN email addresses.

• Remove 'Public bodies with the fewest successful requests'
• Remove 'Public bodies with most overdue requests'

In general I don't think we should shy away from publishing "negative" statistics, but I don't disagree with the proposals based on the background reasoning.

[garethrees]

These statistics need some love… They're not going to get this in the near term.

I think we should make decision based on this basis, but I should note for wider context that we do have some budget from SPF/Adessium projects for research-type work around FOI compliance. I don't want to make any commitments as we haven't scoped that work, but it'll no doubt influence some of our thinking about how we might be able to better provide automated stats like the above.

[ajparsons]

I'd be comfortable with removing the 'awaiting classification' bit as a good stop-gap improvement and get rid of the disclaimer. Is this a very easy or substantial change?

I have broader concerns about the user generated data (especially when batch requests are concerned), but that's a long term conversation. Ultimately I'd like to work backwards from what our intention is (improving responsiveness to FOI) and work out how data gathered from users can contribute towards that. Basically a lot of our stats are 'we have this information' but need to tie into our strategic objectives better (which isn't just collecting information that's relevant, but leading to other actions).

I withdraw the technical comment, have now noticed the prominent "we're aware of delivery from their mail server" (although this does suggest a chasing prompt when we are otherwise aware we expect an automated reply, again deploying information we in principle 'know' to improve results).

[MattK1234]

This was discussed on the WhatDoTheyKnow team catch-up call tonight. There are a couple of notes in the document.

[ajparsons]

Quick talk with @crowbot : my current thinking is we should try and fix the first bullet point disclaimer as @garethrees suggests above. I will draft a broader plan on how we can best use these statistics for our advocacy goals, which may then give a clearer plan for what we want to do in the long run.