Turing ethical approval process

[crangelsmith]

Summary

After discussing with @Ismael-KG we decided to go ahead with the Turing Ethics approval for the TDS project. In this approval from we have to:

• Describe in much details the ideas behind the project (focusing on the pedagogical aspect of the stories, analogy with the DSG)
• Describe how we will carry out the internal ethical approval in the peer review process
• Highlight that we will have to build trust system on our reviewers, and that our internal review process will make contributors thing about these questions in their data science projects (big plus for TDS)

Ismael’s secret guide to answering the EAC form: https://hackmd.io/DneAaMlYSnaPlsHqzskwDQ

What needs to be done?

• [x] We need to build ethics into our peer review process
• [x] Fill in the form for the Turing ethical approval. Use this hackmd for drafting, later we can move it to the word format https://hackmd.io/cVbTkfnoTa-L0HA6QdCTYA

Who can help?

• Everyone

---

Updates

• We created our own ethical framework within our review process, this can be found in our submission and review guidelines
• A first draft of the Turing application form is finished, we are now asking @ismael-KG for comments on the content.

[crangelsmith]

Notes from @samvanstroud

Adding Ethics to TDS

five safes: https://www.ukdataservice.ac.uk/manage-data/legal-ethical/access-control/five-safes

We can use the "five safes" framework as a reference, though it is not directly applicable. We can also use the questions in the EAC form and make sure these are represented in our review process (since our review process is effectively re-implementing the EAC form)

We should also make ethical considerations known at the beginning of story-writing, so that someone doesn't end up writing a story only to be told that it is not publishable.

Answering the EAC form

slack link to form: https://turingdatastories.slack.com/files/U01C263RDMF/F01JEDDRC5C/eag_application_practice_form.docx

[crangelsmith]

A comment from Kirstie in the slack to keep track of:

@Camila - if you can incorporate SUM and FAST that would be really great - they were developed here at the Institute and I think they should be a very useful frame work for you to build a process around. I'm worried that the document will make you feel like you (and the team) have to do huge amounts of work. I don't think that's true. The principles are primarily directed at companies and government algorithms. The Data Stories projects are not likely to carry anywhere near the harms associated with those implementations so you probably just need a lightweight implementation with an escalation process for ideas that aren't immediately easy to approve!

[crangelsmith]

Notes from meeting on the 20th of January:

• @crangelsmith to start drafting something on the application (start with the simple project description just to keep this going).
• We should all think about what will we put in our guidelines, any ideas that come to mind we can just add them as comment to this issue to keep track of them:

For example, today we discussed a couple of ideas:

1. We are focusing on the educational side of the material we are producing, so we can allow contributors to explore different ideas but we should be careful in allowing conclusions that can be controversial (go on the conservative side here).
2. We hope that between the authors and reviewers all ethical concerns can be solved with discussion and suggestions on changing the narrative/conclusions.
3. If reviewers and authors do not agree in a statement or analysis due to ethical concerns we could bring a third party to solve (who will be the third party?).

[samvanstroud]

Coming up with our ethical review process: https://hackmd.io/W-ZGyC-GQbCnMEcwRM4xlw

[crangelsmith]

Notes from meeting on 27th Jan:

Today @DavidBeavan , @samvanstroud and @crangelsmith gave it a go in writing the ethical guidelines for the submission of a TDS, in this document there is a first draft.

Next week we will continue working on the draft. Things that we need to do:

• [x] Finish the guidelines, add discussion in how to deal with disagreements between reviewers and authors
• [x] Add the SUM values to our COC
• [x] Keep working on the Turing form (this should refer to the guidelines we are creating)

[DavidBeavan]

These published ethical approval requests are far bigger in scope that Turing Data Stories, but may be useful:

• Creative Informatics Research Ethics Overview
• Creative Informatics Ethics Statement

[kevinxufs]

Hi @crangelsmith @DavidBeavan @samvanstroud great work with the initial draft for the ethical approval. I think it is shaping up nicely and I just had a few small comments. They are mostly targeted at the following to check boxes:

The proposed data story aims to serve the public good and/or promote positive change and follow the SUM Values described in our code of conduct…

- [ ] Could conducting or promoting this data story create unintended negative outcomes, such as reputational harm?

I think we should be a bit more explicit here as to what we think are positive and negative outcomes. One might find 'public good' and 'positive' to be a bit vague and subjective. For example, perhaps to me just making true things known might constitute a 'public good', but one might instead think 'public good' involves focusing our stories specifically on making known injustices (e.g. Covid deprivation link). Similarly, I feel like reputational damage is one the lesser ethical concerns. Instead what we're presumably worried about are people taking our stories out of context and using it drive forward some sort of potentially discriminatory agenda. It gets messy here because we want to balanace our objective analysis with political considerations of how people might respond to them. There's a sense in which we also want to remain politically impartial in our analysis.

I suspect our best approach is to take a somewhat dogmatic stand here.

E.g.

Stories need to be truthful and clear about limitations of analysis (and potential biases in data). Reviewers should consider if there any obvious ways a story might be construed that would lead to negative social outcomes such as worsening discrimination, injustice, environment, personal attacks etc.

[kevinxufs]

@crangelsmith @DavidBeavan @samvanstroud

Many thanks all for the first draft on this document

Overall I think it looks really good. I've made some very slight wording changes and fixed a few typos, but the content is great.

I had a small question regarding section 5, under the heading: Where you identify risks, you should inform us how your research plans to minimise or eliminate them.

There are a number of questions here - were these questions part of the approval process, or were they suggested by us? If it is the latter, I think we just need a line saying Reviewers will be expected to consider the following kinds of ethical questions as part of their review process

[crangelsmith]

Hi @Ismael-KG, we have a first complete draft of the application form in this document (we plan to move all this text into the actual word document once we are happy with the content).

Could you give us your feedback on this draft before we submit the application? Comments on this issue or in the hackmd document are good. Thanks for your help :)