Farewell [restructure][draft]

[moriarty]

Main Goal

All guests have left the party with their belongings.

Setup

There are guests attending the party and which previously gave their belongings to the robot. See #495. But for the purpose of this task, it is not required that the robot memorize which belongings belong to each guest, they should be able to solve the task using natural language to describe the item.

[kyordhel]

@moriarty Giving object description to retrieve belongings sounds way too hard to me (at least for Stage I). Main idea was just retrieve the coat (by color), but we want to skip manipulation in all Party Host scenario because House Holder is basically manipulation only.

I aso had something else in mind and so is reflected in the commit 736b26e.

[awesomebytes]

First, let me copy-paste here a version of the test:

5.3 Farewell [Party Host]

It’s raining outside. The robot takes tired guests to their cab/uber. Ladies go first.

5.3.1 Main Goal
Some guests are tired, so they call the robot to retrieve their coat. It’s raining outside and there is only one umbrella, so the robot takes the guests one by one to their cab and returns with the umbrella.
Reward: 600pts (300pts per guest).

5.3.2 Bonus rewards
Delivering the right coat (150pts each, max 300pts)
Identifying the female calling person and escorting her first (100pts)

5.3.3 Setup
Location: The test takes place inside and outside the arena. Guests are in the living
room.
Start Location: The robot starts inside the arena at a predefined location near the
entrance door.
Guests: Initially there are five people in the living room, three sitting and two standing. At least two guests are female. All guests have a name and coat color assigned.
Calling guests: Only two guests are willing to leave. Both guests, male and female, will call the robot by waving or shouting at the same time.

5.3.4 Additional rules and remarks
Partial scoring: The main task allows partial scoring. Robot score per guest upon re-entering the arena.
Deus ex Machina: Score reduction for requesting human assistance is applied per guest as follows.
Indicating leaving guest: Telling the robot which guest is leaving (e.g. verbally or pointing at her) causes a reduction of 10% in scoring for that particular guest.
Guiding to the cab location: Guiding the robot where the Cab Driver is standing causes a reduction of 40% in scoring for that particular guest.
Guiding back to the house: Guiding the robot back to the house causes a reduction of 40% in scoring for that particular guest.
Coat handover: Handing over a coat to the robot (i.e. no hook off) causes a reduction of 50% of the scoring bonus.
Kicking out guests: Non-calling guests will refuse to leave the house. The robot must confirm that the guest was calling and willing to leave.
Identifying the Cab: There will be no cab, taxi, or any other vehicle parked outside the arena. The cab location is identifiable by the Cab Driver.
Cab Driver: The cab driver is an unknown person waving, wearing high-visibility clothing (e.g. fluorescent vest), and standing under an open umbrella. The distance between the cab driver and the arena cannot exceed 15m.
Delivering coats: Coats are hanging on a coat rack near the exit door. The robot is supposed to autonomously hook off the coats and bring them back to the right person.

5.3.5 Referee instructions
The referees need to 
 Select at least 5 volunteers and assign them name and a color coat.
 Choose the calling people.
 Relocate the Cab Driver
 Mind the robot when it goes outside the arena.
5.3.6 OC instructions
During Setup days
 Provide coats for training
 Provide umbrella for training
 Provide Cab Driver’s high-visibility clothing for training
2h before test:
 Select and announce the robot starting point
 Help to relocate the Cab Driver

After some discussion we think that in this task we could just drop the "it's raining" and the umbrella references. We see the test being long enough (at least for the 5minutes of Stage I tests) without it. Or move it to Stage II.

[kyordhel]

What about splitting the test?

It must be Stage 1 and have the umbrella/rain because addresses following/guiding and navigation in unknown environments. We really don’t want to leave these features aside. The new feature is person recognition on far distance.

About the coats, that’s irrelevant since other tests already consider manipulation, pairing, and HRI. Nonetheless it fits also well in the party mode.

What do you say?

[nickswalker]

While I agree open-ended natural language descriptions are a significant challenge and that the coats could be excised from the task easily, I think something like the coats should exist in full form in the competition. Now that GPSR is removed from @Home, this could be one of the few places that borders on interesting vision+language.

Reducing it to color detection makes it significantly less interesting:

• From a plausibility perspective, it happens more often than not that most people's coats are a dark shade of only a few colors (e.g. black, blue, green...), so color alone is rarely discriminative. Relative color might be ("the dark green one" or "the light gray") since there are only a handful of coats.
• If teams are going to have access to the coats before hand (which is maybe not representative of real-world conditions), they can address the task as an instance recognition task with existing technology. If they are only grounding color to the instances, there is no challenge. Teams should be encouraged to push a bit further, either via relative attributes (like above), or by grounding a larger set of adjectives (e.g. fur, hooded, trench...). Because they would only apply to instances which the robot has seen before, both of these can still be addressed by a rudimentary parser and hand annotations in a knowledge base, but they are a step closer to the real deal where the descriptions are completely open and the coats have never been seen before.

[kyordhel]

I'm just dropping this. It's related on how to address open-ended NL.

From Commands to Goal-based Dialogs - A Roadmap to Achieve Natural Language Interaction in RoboCup@Home.pdf

[syedaraza]

The tasks are too hard to be done in five minutes.

Navigation in an unknown environments is something which most of the teams could not perform last year, at least in SSPL. Asking to do it twice in five minutes time along with other challenging sub-tasks is too much. However, doing it once will be doable but still challenging.

Also, holding an umbrella and bring it back seems too hard specially for less dexterous robots.

An alternative scenario could be that some of the guests brought bags with them which they want the robot to carry to the cab when leaving.

[syedaraza]

Here is a proposal for an alternate scenario. Rulebook_48-49.pdf

[awesomebytes]

@syedaraza Make a Pull Request with the proposal, it's easier to add comments to it.

[allisonh-tri]

I truly do not understand the emphasis on teaching the robots to identify people by gender and treat them differently based on it in robocup@home.

[kyordhel]

There's no emphasis on it. Is just another feature we included for testing. In real life situations people often refers to other people by clothing, age, hair color, skin color, gender, etc. All these features were included in previous iterations and are included when possible. RoboCup@Home is an international culture-agnostic scientific competition, hence we try to test what people uses.

If this particular feature troubles you, please open an issue requesting its global removal from the competition.

[awesomebytes]

We agreed on the meeting to just make it as neutral as possible. E.g. remove the 'Ladies first' sentence, and make it a decision in the moment if to call first the male or female. Gender recognition is a currently developed and applied perception technique, so I do think it is ok to have it being used.

A pull request with these changes would be appreciated, specially if you want to address it in a specific way.

[syedaraza]

I propose to replace this bonus point, Identifying the female calling person and escorting her first (100pts)

with, Identifying the standing calling person and escorting him/her first (100pts)

[kyordhel]

Identify a standing person is trivial since 2010, and sitting people is not scored since 2014.

Better grant those 100 for a description of the person or a handshake.

[justinhart]

Not everywhere on the planet, but certainly in the US, is it contentious to do anything based on the perceived gender of a person. I get that we can achieve this, but a bigger question looks like this.

rulebook_issues_496.pdf

If you can already accomplish gender recognition then it's of low research value. Nobody can publish the result.

If even in some countries it is politically bad, or could lead to a bad newspaper article, then it is potentially damaging to the league.

So, why even bother with this? Instead of going, "It doesn't bother me," go, "It bothers some people, and that has the potential to overshadow its value in this competition. Even one poor review of our competition will harm our league."

The whole gender thing seems to sit in Quadrant A. I can't publish anything that comes out of it. It has the potential to give our league a bad image. We want all of our tasks in Quadrant D.

[justinhart]

Another way to put it might be this.

If you go to Matt Taylor's Wikipedia page, there's a whole section called "Shirt Controversy."

In a few years, I sincerely hope that someone says, "You know, Justin Hart is missing his Wikipedia page. His research is amazing." There would be a section on HRI, a section on self-modeling, one on service robots and the stuff I've been doing at UT.

What would be missing is the section about the controversy over something that is predictable and easily-avoided.

[syedaraza]

I agree with both @kyordhel and @justinhart

Since the main task is already hard, why not just assign those 100pts to the bonus task of delivering the coats which seems hard too?

In this case, the bonus task would become, Delivering the right coat (200pts each, max 400pts)

[kyordhel]

I understand some features are polemic and that its use or misuse is matter of etiquette in certain cultures, or a reason to get people from triggered to beaten or even dead in some others

Let's stay on topic: the Farewell task

So we have a group of people in a party and some are going to leave. We have the following

Constraints

• We don't talk about the [feature]
• Test lasts 5 minutes
• Test should grant 1000 points (500+500)
• Preferably we avoid manipulation
• No points for recognizing standing people
• Focus on people guiding and navigation in the wild
• Max out interaction and consider person recognition

@syedaraza Thanks for the idea, I liked it but not for Stage 2. Coats is another kind of object and we would need to add it to the General Rules. Plus, its identification can be troublesome and grasping is sometimes tricky even to humans. Guiding 2 people already grants 900 points, so we would just need to replace the french-etiquette-based rule with something nice that grants 100 points. New ideas are also as new tasks to replace this one as PR.

@justinhart Stage I is of no research value. We test integration and check that teams are ready to Stage II. If teams can publish something out of it: AWESOME, we love that, but it is important to point out that there are countless papers stressing that industry is relying on research 10+ years old because is mature, well studied, and now can be used in real world conditions. Publishable results often only work on lab conditions or in MatLab, and won't work in the @Home arena until they are of no scientific value anymore. Unlike journals, RoboCup@Home wants to see stuff running in a non-standardized scenario and that's something that shouldn't change. I'd rather like that peer-reviewers valued more our work of making paper's stuff happen. Keep the big challenges for Stage II

Now, use your awesome TC powers to redesign this test. Add the features you like, we need a fix

[justinhart]

I'll give an honest try at a fix, but I think that we need to emphasize stuff that produces publishable research. I seriously can't have students working on things that don't lead to papers.

[kyordhel]

@RoboCupAtHome/technical-commmittee Can we consider the Farewell task as complete?

Can we close the issue?