search

mysociety / alaveteli

Provide a Freedom of Information request system for your jurisdiction
https://alaveteli.org
Other
389 stars 195 forks source link

Map out what questions we'd need to ask to check an authority has used specific exemptions correctly #5912

Closed garethrees closed 3 years ago

garethrees commented 3 years ago

In order to provide a support checklist to help users challenge refusals (https://github.com/mysociety/alaveteli/issues/5910) we need to understand what questions we can ask them.

We've got some general questions that can be asked about whether the authority refused correctly, but we haven't mapped out what we'd ask for each specific exemption a user may receive.

We'll at least want to consider whether a Public Interest Test is needed, and if so whether it was completed, but there might be other questions to consider for each exemption.

(I've now reordered these in accordance with Graeme's analysis of how commonly they've been applied, below - Myf)

MyfanwyNixon commented 3 years ago

I've found the document where we were compiling notes for the series of blog posts about exemptions. This section where we bundled together similar 'themes' might still be useful, and also NB the stuff about absolute vs qualified exemptions and prejudice-based vs class-based.

MyfanwyNixon commented 3 years ago

Here is a document where I have worked through exemption 12, the 'appropriate cost limit': https://docs.google.com/document/d/1oMuhEONlVmTJRDjonwsqi3VVp5xBDxhtkhc4sroRpU8/edit#

Before I progress any further, can you tell me whether this is the level of detail you are expecting? As I understand it:

This ticket only asks for the questions, but I think it is useful, if already researching into each exemption, to also start compiling the information you'd include in the relevant blog post.

There's also the question of how many contingencies we want to go into for each exemption. Section 12 is relatively simple, but there are six main questions, each with the potential for further variables. Some are more likely than others, but it makes sense, I suppose, to include them all?

Anyway see what you think. I won't do any more work on this until I've had some feedback. Thanks!

garethrees commented 3 years ago

Here is a document where I have worked through exemption 12, the 'appropriate cost limit': https://docs.google.com/document/d/1oMuhEONlVmTJRDjonwsqi3VVp5xBDxhtkhc4sroRpU8/edit#

To my eyes, this looks fantastic! 🙇

Before I progress any further, can you tell me whether this is the level of detail you are expecting?

Worth @wrightmartin, @zarino & @gbp weighing in on this though.

I guess one thing that stuck out at me is the IF YES, BUT item. My suspicion is that this case would be harder to model in code, so we might want to see if we can keep the options to a binary YES / NO. I expect we'll want to do a few more exemptions before making a call on that to see how much it comes up.

As I understand it:

  • They'll be given links to a useful blog post and case studies where we have them

I think there are two options for us to consider here.

  1. We always show a random selection of generic "people battling exemptions" case studies
  2. We try to map specific case studies to specific exemptions

I don't know off-hand how many we have, but my guess is that we don't have enough to fulfil option 2, and option 1 would be the easiest to implement.

  • They may be presented with a template that is pre-populated with useful phrases depending on their responses to the questions.

Essentially yes. I was thinking that the template would come at the next screen (after they click something like "Yes, I'll challenge this!"), but this has made me think of the wob-generator (source code) and how it builds a template from distinct phrases. I don't know which is a better approach from all concerns (ease of implementation, internationalisation, user experience, etc), but certainly something to consider.

This ticket only asks for the questions, but I think it is useful, if already researching into each exemption, to also start compiling the information you'd include in the relevant blog post.

👍

There's also the question of how many contingencies we want to go into for each exemption. Section 12 is relatively simple, but there are six main questions, each with the potential for further variables. Some are more likely than others, but it makes sense, I suppose, to include them all?

I think we can order our questions by "most likely situation", and 6 doesn't seem like too many.

I guess an approach we could take is to get a broad (but shallow) coverage of all exemptions first, then go deep on the most common ones that we see to cover more situations. We can always add to the less frequently used exemptions over time.

MyfanwyNixon commented 3 years ago

I guess one thing that stuck out at me is the IF YES, BUT item.

Having looked at this again, this particular instance could easily be presented as a YES - and now that I know this, I can try to fit all future ones in to a YES/NO binary.

I don't know off-hand how many [case studies] we have, but my guess is that we don't have enough to fulfil option 2, and option 1 would be the easiest to implement.

I agree, we certainly don't have a case study for every type of exemption and in fact, the exemptions are often not really highlighted in the ones we do have (more common to just say 'they refused and we challenged').

We might consider, if time allows, searching on fragments of exemption text to find examples of challenges, then writing them up retrospectively. Perhaps outside the bounds of this piece of work, but it fits in generally with the sort of stuff I'd be doing regularly anyway.

zarino commented 3 years ago

Before I progress any further, can you tell me whether this is the level of detail you are expecting?

Worth @wrightmartin, @zarino & @gbp weighing in on this though.

Looks good to me! I‘ve left a few comments on the doc.

MyfanwyNixon commented 3 years ago

Here's a first draft for Exemption 14 (1) and (2): https://docs.google.com/document/d/1Ev7fujJQ3dP6mBoRBeeJi12SnQf_gEjRtWfgCHM5O18/edit

garethrees commented 3 years ago

Here's a first draft for Exemption 14 (1) and (2)

Parsing out the sub-sections of exemptions feels like it will be pretty difficult to do reliably.

From a content POV, how critical does it feel to treat these as separate Things?

wrightmartin commented 3 years ago

I've taken a look, on paper looks good but I'm wondering if this would be worth prototyping into an actual quiz with Google Forms or something? Might help picture the flow better

garethrees commented 3 years ago

I've taken a look, on paper looks good but I'm wondering if this would be worth prototyping into an actual quiz with Google Forms or something?

I did try to do this when initially looking at how we might tackle this part of the project. It didn't work that well though, because forms assumes a multi-page form when choosing new questions based on a past answer.

If prototyping helps you understand/check what we need to build then thats fine, but I think we'd get more value from prototyping a raw HTML/JS version on e.g. jsfiddle.

Might help picture the flow better

Just to be clear, the flow must all be one a single page (see https://github.com/mysociety/alaveteli/issues/5910).

I'm going to try to spike out the data structure this week, but I'm imagining it will be something like:

section_12:
  - question: A top-level question
      yes:
        action: Appeal!
        followup_template_tags: did_not_confirm_or_deny
      no:
        - question: Another question?
            yes:
              action: "Nothing we can do soz"
            no:
              action: Appeal!
  - question: Another top-level question
  # …

I think we can render this as html and have some generic javascript to expand/collapse levels as they get answered, and then when we reach a leaf node of the tree we extract the template tags that we'll pass to the next step of the flow.

EDIT:

Maybe at the leafs our "action" key is either text or partial so that we can either render a simple message, or use a partial if we want to do something a bit fancier.

MyfanwyNixon commented 3 years ago

Just a note while it occurs to me: we may also want a provision for cases where a request has been turned down but no exemption has been cited. Section 17 of the Act says the authority must:

  1. specify the exemption in question, and
  2. state (if that would not otherwise be apparent) why the exemption applies.
garethrees commented 3 years ago

I'm going to try to spike out the data structure this week

I've now done this https://github.com/mysociety/alaveteli/issues/5910#issuecomment-709076907.

MyfanwyNixon commented 3 years ago

Section 21 exemption: https://docs.google.com/document/d/1ew-pCDxPlT0mYLI0k1koUcEJASEnSk1LLrMM96BWJBk/edit# Section 11 and 11(1A): https://docs.google.com/document/d/1tqwtDUAPB8Nie0ywg65SbiWqHSN0lL87sKbUJn7knR4/edit# Section 40 : https://docs.google.com/document/d/1qSDSz7fG9S9_eNFpwbHhRMbistKLIvQ79nNP1UR40RE/edit# The one I am least confident about thus far, only because it is very compe3lx and the guidance leads off to several other pieces of guidance, mostly around GDPR.

I'll add all subsequent documents to this comment, just to avoid cluttering up the thread.

skenaja commented 3 years ago

Hi Myf

Long time no speak :-)

Just a note while it occurs to me: we may also want a provision for cases where a request has been turned down but no exemption has been cited. Section 17 of the Act says the authority must:

  1. specify the exemption in question, and
  2. state (if that would not otherwise be apparent) why the exemption applies.

Here's a fuller list of this sort of thing (specifically for FOI Act 2000)

Reason for complaint FOI section
A demand for fees was incorrectly applied, or the basis for calculation was incorrect s9, s11B
The recorded information requested does exist and the authority claims otherwise s1
They claim not to be a public authority, or the information requested falls outside the specific classes of info that they are covered for (eg BBC's FOI derogation for journalistic or artistic records) s3-s6
The request was unfairly deemed to be vexatious or repeated s14(1), s14(2)
It was an invalid FOI or EIR request (eg if requester didn't provide their name or state what they wanted clearly enough) s8
They didn't reply in the allowed time s10
They didn't provide "advice or assistance" when it was requested s16
They didn't take any of the requester's reasonable preferences into account for how they wanted the information to be provided s11
They provided the wrong information, or information that was not at the level of detail requested. s1
They're not releasing a dataset in a reusable format s11A
They didn't provide the requested information from their publication scheme, or they don't have a publication scheme s19
The refusal notice they provided was missing, defective/incomplete s17
They're saying it would cost too much/take too much time to provide the information s12
They haven't passed on the request to the Public Record Office for info contained in archived records s15
any exemptions used to withhold information were incorrectly applied s21-s44

Perhaps more a question on how to organise the taxonomy for the fragments of help, some will be generic and will apply to multiple exemptions, e.g. the prejudice test. Many fragments could potentially be mapped to the other laws (EIR, FOISA, etc), however some will need to be tailored specifically for the local law.

I'm happy to provide or review any content for this task, and will request access to the google docs if that's OK.

MyfanwyNixon commented 3 years ago

Hey @skenaja ! Thanks a lot for this. Do go ahead and request access to the docs, though bear in mind they're still pretty much works in progress at the mo.

skenaja commented 3 years ago

Have done, cheers! I've also resurrected https://foiwiki.com (which had died when I switched hosting) in case the content might be of use. Bear in mind that it's not been updated for a long while!

garethrees commented 3 years ago

Here's a fuller list of this sort of thing (specifically for FOI Act 2000) … Have done, cheers! I've also resurrected https://foiwiki.com

Wow, this is awesome! Thanks so much! 🍻

Many fragments could potentially be mapped to the other laws (EIR, FOISA, etc), however some will need to be tailored specifically for the local law.

I think our first pass at this feature may miss some of the subtleties around this as we're contending with time constraints whilst also making it generalisable for other Alavetelis, but always worth these getting pointed out so that we can try to accommodate, or at least note for future development.

MyfanwyNixon commented 3 years ago

Hoping to spend some more time on this today. Question for @garethrees /@zarino /all - is it more useful for me to double back and refine/answer comments on the ones I've already done, or to press ahead creating lists for more exemptions?

garethrees commented 3 years ago

We're still figuring out how we'll actually present all this, so for now I think it's better to rough out some more exemptions, and then refine when we settle on a presentation format.

I've added a list to the top of the issue so that we can track where we've drafted some advice. Might be worth trying to roughly prioritise and move the obviously edge-case ones to the bottom of the list so that we focus on the most commonly experienced. I'm not sure e.g. "Communications With Her Majesty" gets wheeled out that often 👑 !

MyfanwyNixon commented 3 years ago

Thanks @garethrees, that suits me!

skenaja commented 3 years ago

Can a "general" doc be added to the list in the top of the issue so that the request-level issues can be added? I've had to challenge an authority on s17 before :-)

MyfanwyNixon commented 3 years ago

@skenaja The trouble is that (correct me if I'm wrong, Gareth), the software will scan the refusal to see which exemptions it cites - not whether it hasn't cited one at all. Perhaps there could be a case where it scans and finds nothing and the user then clicks to confirm that no exemption has been cited, and then you'd get the s17 advice.

garethrees commented 3 years ago

I think @skenaja's suggestion is valid. If we automatically identify exemptions then we can shortcut some of the process, but we're working on the basis that we definitely won't catch all of them so we should always be able to provide advice, even if we can't spot anything. So, if we don't spot any, we'll start from the beginning – did they confirm or deny, did they cite an exemption, if so, which one/s…

gbp commented 3 years ago

I've used the code from #5931 on the 10000 most recent info requests classified as a refusal on WDTK in an attempt to see the most common legislation referenced:

=> {"Section 12"=>371,
 "Section 1"=>297,
 "Section 14"=>288,
 "Section 17"=>279,
 "Section 40"=>266,
 "Section 21"=>169,
 "Section 31"=>137,
 "Section 16"=>127,
 "Section 43"=>110,
 "Section 50"=>96,
 "Section 8"=>95,
 "Section 10"=>85,
 "Section 30"=>84,
 "Section 38"=>78,
 "Section 2"=>60,
 "Section 41"=>60,
 "Section 22"=>53,
 "Section 35"=>49,
 "Section 44"=>44,
 "Section 36"=>38,
 "Section 5"=>34
....
more... ```ruby => {"Regulation 12"=>34, "Regulation 4"=>33, "Section 58"=>32, "Section 42"=>26, "Section 23"=>24, "Paragraph 3"=>24, "Regulation 5"=>23, "Section 3"=>22, "Section 33"=>21, "Section 7"=>19, "Section 24"=>19, "Section 27"=>19, "Section 2010"=>16, "Paragraph 2"=>16, "Section 32"=>16, "Section 34"=>16, "Section 98"=>15, "Section 20"=>13, "Section 66"=>13, "Section 45"=>13, "Paragraph 10"=>12, "Section 13"=>11, "Section 84"=>10, "Paragraph 6"=>10, "Section 4"=>10, "Section 25"=>10, "Section 39"=>9, "Section 11"=>9, "Section 9"=>8, "Section 18"=>7, "Section 60"=>7, "Section 29"=>7, "Regulation 10"=>7, "Section 70"=>6, "Paragraph 4"=>6, "Section 6"=>6, "Section 47"=>5, "Section 19"=>5, "Section 2009"=>5, "Section 106"=>5, "Article 6"=>5, "Section 80"=>5, "Article 8"=>5, "Section 15"=>5, "Section 103"=>4, "Paragraph 8"=>4, "Section 65"=>4, "Section 37"=>4, "Section 271"=>3, "Section 59"=>3, "Section 100"=>3, "Section 52"=>3, "Section 26"=>3, "Paragraph 9"=>3, "Section 28"=>3, "Paragraph 33"=>3, "Regulation 11"=>3, "Regulation 6"=>3, "Regulation 2"=>3, "Section 348"=>3, "Section 63"=>3, "Regulation 13"=>3, "Section 1997"=>2, "Paragraph 71"=>2, "Section 04"=>2, "Paragraph 7"=>2, "Section 83"=>2, "Paragraph 5"=>2, "Article 3"=>2, "Section 09"=>2, "Section 74"=>2, "Regulation 9"=>2} ```
MyfanwyNixon commented 3 years ago

Brilliant, thanks @gbp!

skenaja commented 3 years ago

I've made a start on general/request-level advice, and some of generic advice applying to multiple exemptions:

https://docs.google.com/document/d/1z17yPku2nCc_EoKKP6eSQuhBHG4K76bOeKuVFNti2uA/edit?ts=5f982389#

MyfanwyNixon commented 3 years ago

In today's catch-up we decided:

@skenaja Thanks for all your input so far - really helpful. I've linked to subsequent documents from the top comment in this thread, in case you want to look through those as well.

MyfanwyNixon commented 3 years ago

I've finished the shorter pieces of advice: https://docs.google.com/document/d/1ijiL2Hx98sySuhPZrNTjBEltHgvQQ33De3pu98Hj1Kg/edit# Next sprint I hope to move on to the comments and amendments made on the existing documents.

MyfanwyNixon commented 3 years ago

This morning I've worked though @skenaja 's more generic advice at https://docs.google.com/document/d/1z17yPku2nCc_EoKKP6eSQuhBHG4K76bOeKuVFNti2uA/edit#

However when it comes to the stuff about public interest tests etc, I think these are covered in the documents for each separate exemption, so I stopped at that point.

That said, they're perhaps easier to access in this format? That is to say 'this exemption is one that does (or does not) require a public interest test to be held). I guess we'll see. We can always copy some of this text across to sit inline with the existing advice and questions.

garethrees commented 3 years ago

I'm going to close this since we've done the majority of the work here.

Once we've configured what we have in WDTK (https://github.com/mysociety/alaveteli/issues/6004) we can create separate issues (referencing back here if necessary) for anything we feel could be improved.

🎊

garethrees commented 2 years ago

Just adding a screenshot from one of the documents where we extracted the text-heavy ICO advice into more of a decision tree, before then converting to YAML:

Screenshot 2021-12-16 at 10 25 31

garethrees commented 2 years ago

For future readers to get a quick idea of the scale of what we did here… for each exemption we compressed ~20 pages of advice into ~5 pages of decision tree to import into the wizard.

exemptions-content