This test evaluates Human-Robot Interaction and the integration of the abilities of the robot tested in stage I. In this test the robot has to solve a task upon request. That is, the test is not incorporated into a (predefined) story and there is neither a predefined set of actions. The task that is to be carried out by the robot is randomly generated by the referees and is composed by 3 subtasks which include navigation, human-robot interaction and robot-object interaction.
The task given by a speech command is composed of three actions, which the robot has to show it has recognized. The robot may repeat the understood command and ask for confirmation. If it can't recognize the command correctly, it can also ask the speaker to repeat the complete command, or ask for further information.
The actual command text may directly reference two actions while a third is inferred by either of the two directly stated commands.
Often, this concerns navigation to the place a command must be carried out.
Focus
This test particularly focuses on the following aspects:
No predefined order of actions to carry out (to get away from state machine-like behavior programming).
Increased complexity in speech recognition (possible commands are less restricted in both actions/operators and arguments/objects, commands can include multiple objects, e.g., put the apple on the kitchen table.
Task
1. Entering and command retrieval:
The robot enters the arena and drives to a designated position where it has to wait for further commands.
2. Command generation:
A command is generated randomly, depending on the command category chosen by the team (see below).
3. Command categories:
All possible actions has been classified previously by the TC according to their difficulty level. The team may choose from the following three categories:
3.1 Category I:
Tasks with a low degree of difficulty (easy to solve). This category includes indoor navigation, grasping known objects, answering questions (from the predefined set of questions), etc.
3.2 Category II:
Tasks with a moderate degree of difficulty. This category includes following a human, indoor navigation in crowded environments, recognizing \& grasping alike objects, find a calling person (waving or shouting), etc.
3.3 Category III:
The same tasks as in category II. However, the information given to the robot will be incomplete or incorrect, meaning that the command as it is specified exactly is not possible. The robot must come up with an appropriate solution to meet the operators' command. Please see the Command examples below.
4. Task assignment:
The robot is given the command by the operator and may directly start to work on the task assignment. The robot must must prove it has understood the given command by repeating it (Please see the remarks about this in section~\ref{sec:gpsr_remarks}).
5. Exiting the arena:
After accomplishing the assigned task, the robot has to leave the arena.
Tasks and actions
The task is composed of three actions, which the robot has to show it has recognized. The robot may repeat the understood command and ask for confirmation. If it can't recognize the command correctly, it can also ask the speaker to repeat the complete command. If the robot fails to understand the given commands, it may ask to the operator to repeat them up to three times, if it fails the team may opt to use the Continue rule (Section \refsec{rule:asrcontinue}). In case the robot has understood the command only partially, it may ask the operator for additional information (e.g.~\quotes{did you say apple juice or pineapple juice?}).
The robot is certainly not required to repeat the command word by word; reprasing the command is allowed.
Required in this test are:
abilities from stage I forming a set of actions $A$ (e.g., following a person, finding a random person, finding a person after memorizing her; finding, recognizing, grasping, and delivering objects, etc.),
a set of people $P$,
a set of questions $Q$,
a set of objects $O$ (the same set as used as in the other tests),
a set of locations $L$ (the same set as used as in the other tests).
Each task assignment contains an action a in A and, depending on the respective action an object o in O, a location l in L, a question q in Q, a person p in P to interact with, or a combination of those. The set of actions is not given beforehand, instead, teams should identify the abilities from Stage I by themselves (and find synonyms for that). That is, L, O and Q are known in advance (provided during setup days), but A has to be found out by the teams (e.g.~taken from freely available ontologies, synonym searches etc.). For the actions A are going to be used common synonyms (like go to}, move to, drive to, and navigate to describe navigation). For the people P, any person willing to operate the robot in a natural way can be expected, however, Professional Operators} are more likely to be used.
Command examples
1. Category I
Go to the bedroom, find a person and tell the time (there is only one person in the bedroom).
Go to the dinner-table, grasp the crackers, and take them to the side-table.
Bring a coke to the person in the living room and answer him a question (there is only one person in the bedroom).
Go to the door, ask the person there for her name and tell it to me.
2. Category II
Go to the bedroom, find the waving person and tell the time (there is more than two people in the bedroom, only one waving).
Go to the kitchen, find a person and follow her (there is only one person in the kitchen).
Go to the side-table, grasp the coke, and take it to the dinner table (the way to the bedroom is crowded and the access to the side-table may be blocked by a human).
Go to the dinner-table, grasp the banana, and take it to the side-table.
3. Category III
Take the apple from the sink and carry it to me.
(There may no apple in the sink, but maybe another fruit the operator might want, an apple may be found somewhere else, etc.)
Go to the kitchen, grasp the coke, and take it to the side-table.
(There may no coke in the kitchen but some other drink.)
Go to the bathroom, grasp the soap, and take it to the side-table.
(The door to the bathroom may be closed. The robot must open the door, find a soap somewhere else or ask someone to open the door for it.)
Grasp the fanta from the small table and carry it to me.
(When the robot comes back with the drink, I have moved somewhere else and the robot must find me again.)
Additional rules and remarks
1. Referees:
Since the score system in this test involves a subjective evaluation of the robot's behavior, the referees are EC/TC members.
2. Operator:
The person operating the robot is one of the referees (default operator).
If the robot appears to consistently not be able to understand the operator, the referees may ask the team to apply the CONTINUE rule (\refsec{rule:asrcontinue}).
With the custom operator, the team can only score 50\% of the points for the respective command.
3. Repeating the given command:
The robot must show it has understood the given command by stating all the required information to accomplish the task. This doesn't mean the robot must repeat exactly the same given command. For instance, if the robot is instructed to deliver a coke to Mary in the kitchen, the robot may ask: do you want me to go to the kitchen, find Mary and deliver a coke to her? or do you want me to find Mary at the kitchen and give her a coke? since both sentences involve all given information.
4. Asking reasonable questions
The robot is allowed to ask a (one) reasonable question. This is to prevent 'phone dialog' style commanding of the robot.
4.1 Misunderstood information:
When the robot did not understood part of a command or it is unsure of what has been told, it may ask the operator to repeat or clarify without fall into a new attempt. For instance, if the robot is instructed to bring me the apple juice from the kitchen table, a valid question for the robot to ask is did you say apple juice or pineapple juice? without considering it as a new attempt for giving the command.
4.2 Missing information:
When a given command lacks of information required for accomplishing the task, the robot should request for that missing part. For instance, if the robot is instructed to offer a drink to the person at the door, a proper question for the robot to ask is which drink should I deliver to the person at the door? It is also possible that the robot simply confirms the command and fetch a random drink at drinks' location.
5. Following people
5.1 Instruction:
The robot interacts with the operator, not the team. That is, the team is not allowed to briefly instruct the operator.
5.2 Natural walking:
The operator has to walk naturally, i.e., move forward facing forward. The operator is not allowed to walk back, stand still, signal the robot or follow some re-calibration procedure.
5.3 Asking for passage:
The robot is allowed to (gently) ask people to step aside.
5.4 Stopping:
The robot must decide when to stop following a person, either because it was instructed to follow her to a certain location, because it was asked to stop by the operator or because the test time is running out. In any case, the robot should state the reason why it changes its behavior.
Data recording
Please record the following data (See Section 3.4):
General Purpose Service Robot
This test evaluates Human-Robot Interaction and the integration of the abilities of the robot tested in stage I. In this test the robot has to solve a task upon request. That is, the test is not incorporated into a (predefined) story and there is neither a predefined set of actions. The task that is to be carried out by the robot is randomly generated by the referees and is composed by 3 subtasks which include navigation, human-robot interaction and robot-object interaction.
The task given by a speech command is composed of three actions, which the robot has to show it has recognized. The robot may repeat the understood command and ask for confirmation. If it can't recognize the command correctly, it can also ask the speaker to repeat the complete command, or ask for further information. The actual command text may directly reference two actions while a third is inferred by either of the two directly stated commands. Often, this concerns navigation to the place a command must be carried out.
Focus
This test particularly focuses on the following aspects:
Task
1. Entering and command retrieval:
The robot enters the arena and drives to a designated position where it has to wait for further commands.
2. Command generation:
A command is generated randomly, depending on the command category chosen by the team (see below).
3. Command categories:
All possible actions has been classified previously by the TC according to their difficulty level. The team may choose from the following three categories:
3.1 Category I:
Tasks with a low degree of difficulty (easy to solve). This category includes indoor navigation, grasping known objects, answering questions (from the predefined set of questions), etc.
3.2 Category II:
Tasks with a moderate degree of difficulty. This category includes following a human, indoor navigation in crowded environments, recognizing \& grasping alike objects, find a calling person (waving or shouting), etc.
3.3 Category III:
The same tasks as in category II. However, the information given to the robot will be incomplete or incorrect, meaning that the command as it is specified exactly is not possible. The robot must come up with an appropriate solution to meet the operators' command. Please see the Command examples below.
4. Task assignment:
The robot is given the command by the operator and may directly start to work on the task assignment. The robot must must prove it has understood the given command by repeating it (Please see the remarks about this in section~\ref{sec:gpsr_remarks}).
5. Exiting the arena:
After accomplishing the assigned task, the robot has to leave the arena.
Tasks and actions
The task is composed of three actions, which the robot has to show it has recognized. The robot may repeat the understood command and ask for confirmation. If it can't recognize the command correctly, it can also ask the speaker to repeat the complete command. If the robot fails to understand the given commands, it may ask to the operator to repeat them up to three times, if it fails the team may opt to use the Continue rule (Section \refsec{rule:asrcontinue}). In case the robot has understood the command only partially, it may ask the operator for additional information (e.g.~\quotes{did you say apple juice or pineapple juice?}). The robot is certainly not required to repeat the command word by word; reprasing the command is allowed.
Required in this test are:
abilities from stage I forming a set of actions $A$ (e.g., following a person, finding a random person, finding a person after memorizing her; finding, recognizing, grasping, and delivering objects, etc.),
a set of people $P$,
a set of questions $Q$,
a set of objects $O$ (the same set as used as in the other tests),
a set of locations $L$ (the same set as used as in the other tests).
Each task assignment contains an action a in A and, depending on the respective action an object o in O, a location l in L, a question q in Q, a person p in P to interact with, or a combination of those. The set of actions is not given beforehand, instead, teams should identify the abilities from Stage I by themselves (and find synonyms for that). That is, L, O and Q are known in advance (provided during setup days), but A has to be found out by the teams (e.g.~taken from freely available ontologies, synonym searches etc.). For the actions A are going to be used common synonyms (like go to}, move to, drive to, and navigate to describe navigation). For the people P, any person willing to operate the robot in a natural way can be expected, however, Professional Operators} are more likely to be used.
Command examples
1. Category I
2. Category II
3. Category III
Additional rules and remarks
1. Referees:
Since the score system in this test involves a subjective evaluation of the robot's behavior, the referees are EC/TC members.
2. Operator:
3. Repeating the given command:
The robot must show it has understood the given command by stating all the required information to accomplish the task. This doesn't mean the robot must repeat exactly the same given command. For instance, if the robot is instructed to deliver a coke to Mary in the kitchen, the robot may ask: do you want me to go to the kitchen, find Mary and deliver a coke to her? or do you want me to find Mary at the kitchen and give her a coke? since both sentences involve all given information.
4. Asking reasonable questions
The robot is allowed to ask a (one) reasonable question. This is to prevent 'phone dialog' style commanding of the robot.
4.1 Misunderstood information:
When the robot did not understood part of a command or it is unsure of what has been told, it may ask the operator to repeat or clarify without fall into a new attempt. For instance, if the robot is instructed to bring me the apple juice from the kitchen table, a valid question for the robot to ask is did you say apple juice or pineapple juice? without considering it as a new attempt for giving the command.
4.2 Missing information:
When a given command lacks of information required for accomplishing the task, the robot should request for that missing part. For instance, if the robot is instructed to offer a drink to the person at the door, a proper question for the robot to ask is which drink should I deliver to the person at the door? It is also possible that the robot simply confirms the command and fetch a random drink at drinks' location.
5. Following people
5.1 Instruction:
The robot interacts with the operator, not the team. That is, the team is not allowed to briefly instruct the operator.
5.2 Natural walking:
The operator has to walk naturally, i.e., move forward facing forward. The operator is not allowed to walk back, stand still, signal the robot or follow some re-calibration procedure.
5.3 Asking for passage:
The robot is allowed to (gently) ask people to step aside.
5.4 Stopping:
The robot must decide when to stop following a person, either because it was instructed to follow her to a certain location, because it was asked to stop by the operator or because the test time is running out. In any case, the robot should state the reason why it changes its behavior.
Data recording
Please record the following data (See Section 3.4):
Audio
Commands
Images
Plans