Turing Test

[gunthercox]

I would like to allow this program to begin to address the Turing test concept.

A few common ways that users from a study attempted to determine if an entity was a computer or a person included the following:

• Ask name
• Ask gender
• Notice repeated information
• Ask questions repeatedly to see if different answers are given
• Try to get the entity to contradict itself
• Ask math questions
• Length of time taken to respond

I do not plan to immediately address the last two bullets regarding math and speed. The math questions inherently suggest that the chat bot should get some math problems wrong in order to seem more human. My view on this is that it is more useful to have a bot that is good at math, which could just as easily be a human. Also, the amount of time taken to respond is not critical at this point but could be easily added if needed.

A great selection of turing test questions: http://greatbird.com/turing/

[justindomingue]

Heads up: your link is dead

Remember that for the bot to be able to pass the Turing test, it should be able to answer questions like

Explain why the following story is funny/absurd: "A man walks into a bar and say 'Hello, it's me!' - but it wasn't him..."

My point is that although your implementation might pass a restricted Turing test, aiming for the full version will require more semantic analysis.

[gunthercox]

@justindomingue You have a valid point. As I've researched this topic more in-depth I'm beginning to see a trend in the types of questions that are asked. An example might be something such as

"What is the meaning of life?"

which is actually an invalid Turing test question. The problem with it is the assumption that the machine can't define the meaning of life because it isn't alive. In reality, producing a definition to answer a question is very easy to do.

At the moment, I believe that the best questions for a Turing test require the subject to express abstract application of their own knowledge. By this I mean that they have to be able to leverage their own knowledge toward an end result. An example I've been using for this is the following question:

Pretend you are Arnold Schwarzenegger.

This command should invoke some serious cognitive requirements to test the subject. This is because it requires self-recognition, knowledge of an existing subject, and the ability to apply that knowledge onto itself while using it to issue a response.

I'm in the process of researching more possibilities for valid Turing test questions. This is obviously a very long-term goal, but it is useful to have the idea out in the open so that it can be discussed.

[gunthercox]

Just wanted to post a few more thoughts here.

1. Testing awareness of own knowledge. In this case if the entity being tested confirms competency to answer a given question, then it should be able to answer the question when it is asked. This also could be modified to ask a more generalized version that questions if the entity knows how about a general subject, for instance do you know how to compute natural logs?
   • Q: Do you know how to compute the natural log of seven fifths?
   • A: Yes
   • Q: What is the natural log of seven fifths?
   • A: I do not know.
2. Testing creative intelligence. "Make up a story about X." This tests the ability of the entity in question to hypothesize about the nature of a given object or situation. This also demonstrates how well an entity might be able theorize when dealing with incomplete information.
3. Ability to resolve states of cognitive dissonance. A simple way to test this may be to lie to the entity in question at one point in a conversation. By giving it contradictory information where one of the two components is clearly false, the entity should assume the fact to be incorrect.
4. Test grouping of information. Humans have a natural tendency to mentally group similar objects and ideas.
   • Which one of these things is not like the other...
5. Can a concept of self be defined?
   • There is a problem when the intelligence comes from the brilliance of the programmer and not the power of the machine.
   • Emergent order in chaotic systems → Neurons in the brain
   • Test based: That playing a sound causes an activity in x part of the brain.

[navyad]

Turing questions

http://britell.com/misc/turing.html

[gunthercox]

I'm going to close this off as a ticket and move the content of the discussion into the project wiki (https://github.com/gunthercox/ChatterBot/wiki/Turing-Test-Questions). Eventually my goal will be to create a document that proposes a methodology for creating good questions to ask a computer / human to try to determine which is which. Obviously this project is still a ways away from this level of interaction, but this is still a concept that is somewhat interesting.