Bring back Robozoo

[johaq]

I realize that this proposal does not fit with the new GPSR like approach to tests but I would still like to see this discussed.

One problem that was touched upon in Montreal was the problem of scoring social interaction objectively for @home in general and for the SSPL specifically. The solutions to this in current research that I am aware of generally are to collect many subjective views to reach objective conclusions mainly through user studies. This is really not doable in the current Robocup setting imo (handing out questionnaires after tests to the audience etc.). The idea is to have a scenario similar to the old Robozoo as follows:

We have a specified area somewhere outside the arena where every robot is placed when robozoo starts. Audiences can enter the area through an entrance where everyone is handed one or multiple coins. The audience can freely go through the area and interact with the robots. They are instructed to give their coin(s) to the robot(s) that they liked best. This ideally goes on for quite some time, maybe 2 hours. Teams are scored based on how many coins the robot collected. I think before each robot had an assigned space. I would prefer to allow the robots to freely roam the "Zoo Area" and approach people or position themselves smartly. Teams not entirely comfortable with their navigation can still just remain in their space but other teams could have their robot go near the entrance or exit for example or try to approach large groups ideally where not a lot of other robots are already. There also used to be team members accompanying the robot. Since this is about human robot interaction and not teams explaining their robot I would prefer to have no team member in the area after the start signal. Each team gets maybe 2 or 3 jokers where they can remove the robot from the area, restart or fix some issues and then place their robot back inside.

I think this proposal is highly engaging for audiences which is one of the biggest goals for the future (#477) and can bring an aspect of social interaction to @home that can be scored at least somewhat objectively.

[awesomebytes]

My random thoughts on this:

Even tho I do like the idea of people interacting with the robots freely and getting some insight from it... I find very hard to score this in a fair way.

For the SSPL league at least all robots basically look the same (aside from stickers or maybe some clothes and the interface of the tablet differentiating them), for OPL... well I guess having a good looking robot should be compensated someway too?

When kids come to see @Home they always wanna interact/touch/play with the robot and I'd like that to be possible too.

[johaq]

For the SSPL league at least all robots basically look the same (aside from stickers or maybe some clothes and the interface of the tablet differentiating them), for OPL... well I guess having a good looking robot should be compensated someway too?

I agree that I like this idea more in the standard league context.

When kids come to see @Home they always wanna interact/touch/play with the robot and I'd like that to be possible too.

That is exactly what is possible here if a team wants to go for it. I would not put any restrictions what the robot is allowed to do and what is not.

[kyordhel]

RoboZoo was always hard to score and scoring was very unfair. We had situations in which several teams asked their colleagues in other leagues to vote for them. Moreover, there was little to no voting from part of the audience compared with voting from people from other leagues.

What is more, and the most annoying thing in my opinion, it turned into a carnival. Complex manipulation and high level reasoning dimmed against a robo-dancer playing some music and choreography, so most robots went after that.

Therefore, I would set well-defined restrictions on what robots can do and what can't do, but more in a thematic way, like the Demo Challenge. Robots are free to do whatever they want within the scope of the year's theme.

Another restriction would be cap the score of RoboZoo to the maximum score achieved in the stage (like Open Challenge) to prevent team shuffling based on a single test.

[justinhart]

I don't think that this really objectively tests anything of interest.

Let's suppose that this is supposed to demonstrate human-robot interaction. Which robot scores better: The one which flawlessly tracks gaze and performs human-like eye-contact and handover behavior or the one that blinks LEDs and plays whatever song is popular at the moment?

[raphaelmemmesheimer]

I'm against the reanimation of this test. From my point of view this was the most unscientific test that we had in RoboCup@Home. And RC stands for bringing science from the lab into applicative real world scenarios (may my opinion). I understand the social interaction aspect but we should foster this from another perspective. I would prefer to see more test like Restaurant, where robots read intentions of approaching people and act according to it.

[johaq]

We had situations in which several teams asked their colleagues in other leagues to vote for them.

With cloud services allowed we already heavily rely on the goodwill and sportsmanship of teams. I would like to think that this is something the teams would not consider and police each other on (maybe wishful thinking).

Therefore, I would set well-defined restrictions on what robots can do and what can't do, but more in a thematic way, like the Demo Challenge. Robots are free to do whatever they want within the scope of the year's theme.

This seems like a good idea to me. A theme could also give the audience some guidance on what to expect.

Another restriction would be cap the score of RoboZoo to the maximum score achieved in the stage (like Open Challenge) to prevent team shuffling based on a single test.

I did not mention it but I would just set a maximum score and teams get X% of that score if they collected X% of the coins handed out

Which robot scores better: The one which flawlessly tracks gaze and performs human-like eye-contact and handover behavior or the one that blinks LEDs and plays whatever song is popular at the moment?

I think this underestimates what audiences expect from a robot coming to an event like Robocup.

From my point of view this was the most unscientific test that we had in RoboCup@Home.

Agreed. Still...

And RC stands for bringing science from the lab into applicative real world scenarios (may my opinion).

this is the closest to the current real world scenario that Pepper is mostly used in. I'd like to point out that I like this proposal much more in the context of the SSPL than in the other leagues.

I would prefer to see more test like Restaurant, where robots read intentions of approaching people and act according to it.

I agree that this would be nice but the problem remains: How do we score/award this? I dont really expect people to act natural with a referee following the robot everywhere and team members shouting and gesticulating from the sidelines which is the current setting in most tests. Furthermore it is even more unfair if the robot is scored based on one or a few interactions in a test given that performance then just depends on how nice/experienced the users are.

[kyordhel]

@justinhart what we saw in the past was blinking LEDs and trending songs winning the gaze (and the test). @airglow Agreed.