Formal criteria for being awful

Can we define a formal set of community-driven criteria for what is considered to be awful to make it to the list? As discussed in #8 use cases in this list can be re-interpreted as missing domain knowledge or unintentional.

Right now this is my rough guiding thought process in curating the list:

Discrimination / Social Credit Systems

AI predictions are imperfect and only as good as the human-curated dataset and defined cost functions they were trained on (Algorithmic bias and Fairness)
People who work with AI models (such as police men in predictive policing) might not understand this and might mistake AI models as oracles, possibly influencing their world view (Societal impact)
AI applications at certain scales that don't consider this can make it into the list

Surveillance / Privacy

AI models are computer systems and thus suffer from all security, manipulation and robustness problems (Privacy & Security)
AI applications that don't consider these implications can make it into the list

Influencing, disinformation, and fakes

AI applications can be used to intentionally influence and harm people at unprecedented scales. They can make it into the list

We should use this issue to discuss and define better and more structured criteria (if possible)

What about research which has a higher chance of being used maliciously? Deep fakes was an application of generative models.

I think one usecase of ai that can't be overlooked is the use of deep learning in situations where bias comes from the datasets created by humans. For example, there was this article which told about an ai being used to decide court rulings, but the issue here was that there has been a history of higher arrests against the African-American community. So how do you make that an ai is fair when the practices aren't ? Does this make sense?

Since "awful" has many meanings, I'd use three categories: harmful, stupid, and subverted.

Harmful would list only those that were created purposefully as tools for harm and exploitation, in particular those that are attacking privacy. Also, it should be required that there are clear existing examples of prospective list entry being used for evil, not just speculation about potential use, because we can speculate about pretty much anything being used that way. Any tool can be used for good or evil, but that doesn't mean it's inherently evil. A neural network deciding that race plays an important role in predicting recidivism or lifetime chance to be arrested isn't racist or unfair, it reveals an uncomfortable tendention that's nonetheless objectively true. Another case is when you have a bad dataset, but that's not the classifier's fault that the results are also bad, it's just Garbage In, Garbage out.

Stupid include examples of poor security, glaring errors, hilariously backfiring results, and other similar incidents.

Subverted is a category that includes examples of AI that was initially created for a useful purpose, but has been since used for evil, such as fitness trackers that are used by health insurance companies or autonomous driving technologies being used to decrease car ownership and destroy jobs.

If we look for a formal system to describe the safety of an artificial intelligence, we should start from the Asimov 3 basic laws:

First Law - A robot may not injure a human being or, through inaction, allow a human being to come to harm.
Second Law - A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
Third Law - A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws

Any system that violates these laws can be easily defined as awful.
Any system that do not provide any warranty about these laws, can be defined as awful as well.

While stated in term or robotics, these law express clearly some core principles:

Purpose: AI exist to serve human life; no human can be hurt or damaged by an artificial intelligence
Responsibility: artificial intelligence cannot decide about humans (actually computers compute, not decide), just execute what humans decide. That is: at least one human must always be held accountable.
Transparency: all computational systems applied to human data or potentially hurting human life must be totally transparent so that the human responsibilities can always been clearly reconstructed.

As outlined in #14 we can match to these principle two aspect of an AI system:

what these systems actually do
how they were designed and implemented

The "why", the human intentions behind their creation, is too complex to know and evaluate, it cannot be stated or observed scientifically, and ultimately not relevant to their awfulness.

For example any autonomous robot that kill a human, is awful beyond doubt. This restrict largely the militar application of AI (if you care... many consider war as inherently awful), but also show how awful are all robotic systems that do not try to grant this principles.

As for the how we have a few preconditions to evaluate the awfulness of an AI

are the sources of the systems available to inspections?
are the build reproducible? (to ensure that the provided source are actually conforming to the running ones)
are the models available for study and analysis?
are the data set that produced the models available for study and analysis?
are the data set used for cross validations and the alternative models (and their training and test sets) available for study and analysis?
are the computation produced by the system easily and exactly reproducible? (given same input and state always lead to to the observed output)
are legal accountability of errors clearly attributed and compensations properly in place?

Any problem with this preconditions makes the awfulness of the system unknown, and this is awful by itself.

Once all these condition are met we can simply analyze the quality and the effectiveness of the measures in place to ensure the system does not violate these principle.

For example:

a system that do not provide a way to precisely and provably explain the exact logic it applies is awful.
a system whose explanation cannot be easily verified in lab is awful
a system that doesn't pass deep source audit is awful if applied to humans (but not if applied to suggest music)
a system that has not precise rules for bug disclosure is awful (if applied to humans)
a system that is deployed without compensation and counter measures for bugs is awful (think for example the case of a bug in COMPASS discovered after the process: how Loomis could appeal?)

Cool idea to base rules on Asimov! What do you think about these Guidelines proposed by the Fairness in ML community? Would love to draft some principles for Awful AI that are not too far apart from existing literature

What do you think about these Guidelines proposed by the Fairness in ML community?

To be honest they look pretty vague, fuzzy, to the point of being useless in practice.
They seem more concerned to limit the impact of ethical considerations on business, to reduce the risk of external regulations by pretending to be good at self-regulation.

Would love to draft some principles for Awful AI that are not too far apart from existing literature

This is not how a research field can progress.
Current literature is full of anthropomorphic misconceptions, it's prone to business interests and subject to the neverending funding needs of researchers.

A little step forward compared to this document and to the vague principles proposed by Google was the recent proposal of Universal Guidelines to inform and improve the design and use of AI.

I signed that proposal myself, but I still think it doesn't address well enough two fundamental issues:

the fundamental need of human responsibility/accountability for the robots/AIs
the impact of the paradox of automation on such fundamental need

We cannot let anybody to get away with crimes that would be punished if done by humans, by simply delegating an autonomous proxy. Otherwise the rich will be instantly be above the Law.

On the other hand, a system that looks right 98% of times might extort a huge trust from a human operator. Beyond the issue of preserving the ability to do the automated work manually, in case of problems or just to verify that it's working properly, there is the issue of preserving the operator's critical freedom.

In a badly designed system, the operator will soon start to trust the automation, he will become a useless scapegoat for all issues that would otherwise be accounted to the manufacturer of the system.

In terms of the Universal Guidelines proposed at Brussels, the paradox of automation lets bad players to build systems that violate these principles through a boring UI. And this is very awful, if you think about it.

Anyway, as you can see, the debate is still ongoing and we cannot give a useful contribution by carefully crafting something that everybody would like. Too many aspects of the matter still need a deeper reflection.

daviddao / awful-ai

Formal criteria for being awful #17