ethics/ transparency audit

[jwzimmer-zz]

Alpha version of checklist at: https://www.overleaf.com/read/vrqgnmmysrbc

[jwzimmer-zz]

Rough list of potential items for checklist

1. Ethics as part of a project's preliminary Needs Assessment
   • Identify tools needed - any issues with procuring tools, or resources or knowledge required for tools?
   • Identify data needed - any issues with procuring data, or resources or knowledge related to data?
   • Any issue with publishing data? Should it be anonymized? How easy would it be to de-anonymize?
2. Since this project passively analyzes data that already exists (no gathering new data or participants), what is the analogue of informed consent for the people who created the data?
3. How is the outcome of this project going to be disseminated? Who will see it? Who can see it? Who should see it?
4. What is the potential impact of this project?
5. What are the potential harms of this project?
6. What will happen to the data and other work-product generated during this project, once the project is over?
7. For transparency and usability: create a "birth certificate" summary of project
8. Recurring audit for transparency, ethics, etc.
9. Make changes in response to audit
10. Donate money to (1) a social justice cause and (2) an environmental cause

[jwzimmer-zz]

First pass

1. Ethics as part of a project's preliminary Needs Assessment
   • Identify tools needed - any issues with procuring tools, or resources or knowledge required for tools?
   • Identify data needed - any issues with procuring data, or resources or knowledge related to data?
   • Any issue with publishing data? Should it be anonymized? How easy would it be to de-anonymize?

I think since all of the tvtropes site content I have access to is already public, and there is no plan to focus on individual users in any way, there aren't any new risks to those users. I think the tools needed will be mostly open-source, semi-open-source, or provided by UVM.

1. Since this project passively analyzes data that already exists (no gathering new data or participants), what is the analogue of informed consent for the people who created the data?

I think their participation in the site is sufficient consent.

2. How is the outcome of this project going to be disseminated? Who will see it? Who can see it? Who should see it?

I don't know.

3. What is the potential impact of this project?

I don't know.

4. What are the potential harms of this project?

Could cause burden on the tvtropes site by scraping; hopefully that is completely mitigated by the rate limit on the wget process. We could reinforce narratives and tropes by discussing them. We could paint the participants of the site reductively.

5. What will happen to the data and other work-product generated during this project, once the project is over?

I think it will stay indefinitely on github. I don't think that introduces any additional risk.

[jwzimmer-zz]

Going over the questions in the Datasheets for Datasets paper and answering some of them in the interest of transparency... these aren't the most thorough and careful answers ever, but I'd rather have something than nothing as far as describing the repo in an organized fashion. I think reading over this at least gives you an idea of what this is.

Questions below are from https://arxiv.org/abs/1803.09010:

Subjects: | Databases (cs.DB); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: | arXiv:1803.09010 [cs.DB]
  | (or arXiv:1803.09010v7 [cs.DB] for this version)

Re https://github.com/jwzimmer/tv-tropening & https://github.com/jwzimmer/tv-tropes

Section 3.1 Motivations

• For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. We gathered this data - I think it may be too disorganized and chaotic to properly comprise a dataset - because we wanted to try and gain insight into how stories work in our culture via the information in the TV Tropes website. Proximally, we were working on a project for our Data Science class and following the advice of our advisors/ mentors.
• Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Phil Nguyen & Julia Zimmerman on behalf of UVM, Nick Cheney's Stat 287 class and Danforth & Dodd's Computational Story Lab.
• Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. UVM, indirectly, since they provide the relevant classes and support the students and professors mentioned.

Section 3.2 Composition

• What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. Most of the pages we downloaded from the tvtropes site are "trope" pages, describing a trope that appears in media (a movie, a tv show, a comic book, etc.), listing works it has appeared in, and linking to related tropes. We constructed objects that have a network structure, from dictionaries that map a trope page to the tropes that page links to, to gml files and gephi files that model parts of the site as a network whose nodes are tropes and whose edges indicate links within the body of that trope. There are other assorted objects, scripts, and content as well.
• How many instances are there in total (of each type, if appropriate)? I don't know. There are around 27000 trope pages.
• Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable). Not all. We defined "trope" based on https://github.com/jwzimmer/tv-tropes/issues/13#issuecomment-718109481 and intended to capture all of those.
• What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description. There are objects created via python, mostly json and gml files, and charts. The raw data is mostly .html files we downloaded from the tv tropes site (text only).
• Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them. No. Not intended for machine learning.
• Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. Probably! So caveat emptor please.
• Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? I think "self-contained", although there are almost certainly links in the html pages that link to pages we did not download from tv tropes. So maybe not self contained?

Section 3.3 Collection process

• How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. All the data was scraped from the TV Tropes website (https://tvtropes.org/). We used beautiful soup to look through the html for the parts of each page we thought were relevant. We sort-of documented what different things in the repo are here: https://github.com/jwzimmer/tv-tropes/issues/13
• What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? Wget, beautiful soup, we also downloaded some pages manually. Our validation processes were mostly programmatic, looping over things and making sure that output matched what was expected, or manual, comparing our results to what we could see on the site and making sure they looked reasonably similar (https://github.com/jwzimmer/tv-tropes/issues/9, https://github.com/jwzimmer/tv-tropes/issues/12, https://github.com/jwzimmer/tv-tropes/issues/18).
• If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? We intended to get every trope html page on the site that was tagged as page type "trope", but I don't have proof we succeeded.
• Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? _Students (Phil and Julia), with some advice from faculty and staff at UVM. Julia was compensated indirectly as a full-time PhD student (thank you, UVM!).
• Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created. Dates should actually be reliable! We pushed and pulled from the repo as we worked, so timestamps should be pretty accurate!
• Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. We discussed the ethics ourselves, and included that in our final write-up. This was not a formal process. These answers are also a part of an ethical review, kind of, since I (Julia) and going over these questions in an attempt to provide some transparency as an ethical requirement of this project. Not a high bar, but basically I don't want to be a jerk.
• Does the dataset relate to people? If not, you may skip the remainder of the questions in this section. Yes
• Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? From the tvtropes website.
• Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself. No. The website is public; I don't think any user of the site could use it without understanding that, so we didn't think we needed to notify anyone. • Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented. No.

Section 3.4 Preprocessing/cleaning/labeling

• Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section. Sort of, in that we were only looking at certain pages and certain attributes within those pages.
• Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data. Yes, everything we scraped directly from the site is saved in these repos as well.
• Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point. All (or maybe "most", if we left something out unintentionally) the python scripts and notebooks we used are in these repos as well. We mostly used python, wget, and beautiful soup.

Section 3.5 Uses

• Has the dataset been used for any tasks already? If so, please provide a description. Yes, it's been used for a Data Science 1 project, and may be used for continuation of related projects.
• Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point. Yes: https://github.com/jwzimmer/tv-tropening & https://github.com/jwzimmer/tv-tropes
• What (other) tasks could the dataset be used for? I'm not sure.
• Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms? Yes, the nature of the data is that it involves a lot of stereotypes! It's not exactly clear to me how to say when you are definitely referencing or using a trope that has stereotypes in it vs. when, by doing so, you're endorsing it. I don't want to reinforce stereotypes, but I do want to study stories. I hope any future users will feel similarly. • Are there tasks for which the dataset should not be used? If so, please provide a description. Probably lots of tasks for which this data isn't suited. It isn't meant to be used as a basis for machine learning or solving any particular problem.

Section 3.6 Distribution

• Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. No. It's an open repo, so it's public, but I have no plan to intentionally disseminate it or get other people to use it.
• How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? It lives in two repos on GitHub indefinitely: https://github.com/jwzimmer/tv-tropening & https://github.com/jwzimmer/tv-tropes
• Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions. Not as far as I'm concerned. I don't know about the relevant legal things. I wanted to look at a static version of the TV Tropes site and be able to mess with it without being onerous to their infrastructure, that's the reason for making our "dataset".

Section 3.7 Maintenance

• Who is supporting/hosting/maintaining the dataset? As long as we're working on the project, Phil Nguyen and Julia Zimmerman. As long as GitHub lets me keep my repos here simply and for free, also GitHub, indirectly.
• How can the owner/curator/manager of the dataset be contacted (e.g., email address)? jwzimmer1@uvm.edu
• Is there an erratum? If so, please provide a link or other access point. No...
• Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)? Yes, it will be updated as Phil and I work on the project. There are no other users so updates won't be communicated, although they'll be documented on GitHub as the project has been up to this point.
• If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced. No.
• If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description. Well, there is no process for this. If you contact me I will be happy to check out your proposed contributions.

[jwzimmer-zz]

This is still relevant and should be re-visited. What we wrote above was mainly thinking of the tv tropes data, not the character space data, so we should re-visit this issue in that context.