parallel-forecast / AI-dict

Open-source terminology standards for AI forecasting projects.
https://parallel-forecast.github.io/AI-dict/
MIT License
2 stars 2 forks source link

How we should we deal with terms with free variables? #4

Open lagerros opened 5 years ago

lagerros commented 5 years ago

Consider the Open Philanthropy Project's sample definition of "high-level machine intelligence" (HLMI):

A computer system can outperform the median-skilled employee...

  • ...for 95% of occupations listed by the U.S. Bureau of Labor Statistics...
  • ...with 6 months of training or less...
  • ...at 10,000x the cost of the median employee or less

Different choices of input parameters (95%, 6 months, 10000x) can capture very different things,

Option 1

Restrict all future use cases to a particular parameterization.

Option 2

Allow functions in the dictionary, e.g. listing the term as:

HLMI(percentage, months, cost-multiple)

and writing questions as:

By 2029, will we have HLMI(0.95, 6, 10000)? [ai-dict-v1]
davidmanheim commented 4 years ago

I suspect that modularizing the terms might solve some of this, at the (perhaps unacceptable) cost of greater complexity for users.

If someone wants to use different values, they just need to note that they are using terms different than the ones in the dictionary, and provide the alternative definition. This both makes it easier to define, say, Viable Moderate-Level Machine Intelligence, by changing [Almost All] to [a Majority], or to disagree with the dictionary completely about something more basic, such as basing the "Job" category on some other standard than BLS, without modifying every single place it occurs.

In the example of (Viable) HLMI, I've split out the relevant factors below.

Viable HLMI: A [Commercially Viable] [Machine learning system] that makes [Almost All] [Jobs] [Automatable].

Commercially Viable: (For Automation) A [Machine learning system] which costs less than 10,000 the cost of the person / task being replaced, using [available] training data.

Short Time Frame: (Training) Less than six months

Machine Learning System: A system trained on data - including “self-play” rather than having its behavior explicitly programmed

Available: Extant, or able to be created in a [Commercially Viable] fashion in a [Short Time Frame].

Almost All: (Jobs) - 95% of [Jobs]

Job: A job is one of the occupations listed by the U.S. Bureau of Labor Statistics in the 2010 SOC standard.