Describe all challenges in terms of GPSR

[LoyVanBeek]

This issue is for brainstorming only, at least for now.

Instead of describing each challenge in relatively high detail, write challenges in terms of a series of GPSR commands. Each challenge in the 2017 rulebook lists a bunch of requirements for yielding points and for the arena etc. Instead, we can have more general rules for every command and what variations of success there are in it plus potential hurdles in executing a command we might impose. For example:

• Navigate to B: reach location B within manipulation distance while avoiding any obstacles. Expected difficulties are: obstacles in the form of small object, human crossing path, 3D obstacle (...).
• Grasp an item: Hold an item for at least 5 seconds. Points are discounted when the item is damaged, contents are spilled. Difficulties are: objects are not standing upright, standing close together etc.
• Follow: follow the operator along an undefined path both inside and outside the arena. Crossing a door grants 5 additional points, every 10 meters inside grants 10 points, every 10m outside grants 15 points (for example!). Difficulties are: human(s) crossing between operator and robot

Once that is in place, many of the current challenges can be written much more succinctly:

• Help me carry

  • Follow me
  • Remember, here is the car
  • Take this bag
  • Bring the bag to the kitchen and guide a person to the car
  • Done.

• SPR:

  • Tell me about the crowd
  • Answer a question
  • Repeat

• Storing Groceries

  • Tell me what objects are on the table
  • Place the apple near the apple
  • Place the pear near the pear
  • ... (note that the operator gives explicit commands here, while you'd rather do for all objects on table...

• Restaurant & Cocktail party:

  • Go to the waving person and take an order
  • Bring me pizza and beer
  • Repeat

• SATTU:

  • Put a knife, fork, spoon, cup and plate on the table (or specify 'al cutlery' or each piece separately, with varying amounts of points)
  • Wait in the corner
  • Bring knife, fork, spoon, cup and plate in the dishwasher
  • Wipe the table

What's prohibits us from implementing all challenges this way?

[balkce]

I think it will definitely result in a much condensed rule book. But why stop there?

To be fair, most of the Stage I tests are already a sub-set of GPSR, so why not just do GPSR and only GPSR in Stage I? By doing it like this, we could randomize the commands such that all type of functionalities are tested (navigation, manipulation, voice recognition, etc.), and force teams to not script the tests (which is what most do). I'm pretty sure I'm not the first that has proposed this.

To this effect, complex story-based tests are carried out in Stage II. Imagine doing a help me carry with actual bags, or storing groceries by taking the objects from inside the bags. These tests showed a lot of promise, but required simplification since they were Stage I tests. But because we would be doing GPSR so much in Stage I, we could remove EEGPSR from Stage II. We could also throw Open Challenge back into Stage I. This leaves us with two open slots for new tests which could be really interesting, complex and story-based, and not be as scripted as the ones that are in right now (Set up table and Restaurant).

[LoyVanBeek]

I totally agree. But, in first instance, I would like to keep the tasks we have now in place with minor modifications for 2018. This way, the teams can improve the performance of them instead of baking completely new ones, again. We just change the description in the rules, not what actually happens. If we like it, we can go further.

There has always been a reason to not do GPSR only (which indeed has been proposed in the past as well). Describing more or less scripted challenges in terms of GPSR may be a nice middle ground: you can still script things but you can also do it the proper way of actually doing GPSR.

[kyordhel]

I don't totally agree with this. In words of my supervisor (rephrasing)

[...] if I need to explain the robot how to do this, or how to do that step by step, pffffff!!!! I rather do it myself, or hire someone, is cheaper...

So yes, we need more GPSR, but not so explicative. Command should be as complex as store the groceries, receive the guests, set the table, do laundry, clean the bathroom, etc. and robot needs to comply. I would rather say that the rulebook describes a set of constructs or basic tasks, and that all tests are explained in terms of these constructs.

In addition, for GPSR we need to know how people (not us) command robots and what do they expect (in my last poll/experiment, more than 60% want the robot to walk the dog and shovel/clean its crap).

[LoyVanBeek]

Again, I agree but giving more detailed commands is an acceptable first step. We might even go so far to give more points if we need to give less commands and thus reward being autonomous for a longer time.

For now, I don't care where the script of the challenge is stored, either it's in the rulebook as a story of the robot looks it up internally based on the command (at least it's that easy in theory).

[balkce]

@LoyVanBeek I was thinking more in long term. I agree that the 2018 rule book needs to be a variation of this year's rule book.

@kyordhel Actually, that would be a good evolution of the current GPSR (new Category maybe?): implicit commands. "I'm thirsty, robot." should be enough to trigger the robot to fetch a drink. I would definitely like to push for robots to respond to natural wording that ask for a complex task, instead of forcing the user to talk like a robot.

Coming back to the issue of this thread: I'm fine with re-writing the tests based on explicit/primitive functions. However, I'm wary that it would be a whole lot of work to land on basically the same rule book. Unless you guys are up for the task to do it this year (hell, I may even jump in if the other TC members ignore their job as they usually do), I think it will be a better use of our time to do this jump when we need to write a new version the rule book (2019?).

[kyordhel]

@balkce I won't add a new category. Splitting EEGPSR didn't work and the next step is merging it back. For GPSR having categories mapped to difficulty degree seems to work better, so implicit commands should be incorporated to Cat 2 & 3 so robot can get either implicit or explicit command.

Regarding rewriting the rulebook in terms of primitives... You need first to convince me is not a dead end. I support goal driven tests, not checklist.

[balkce]

@kyordhel fine.

As for the issue of re-writing, I come back to what I said before:

I'm wary that it would be a whole lot of work to land on basically the same rule book.

[LoyVanBeek]

On the robot behaviors: yes. But I see it as a refactoring to build a more extensible foundation for steps after the refactor, such as those suggested above.

The challenge descriptions themselves can be as short as the scoresheet: Just list he commands instead of the \scoreheading{...} we have now with more explicity listed gradations in achieving a command.

[balkce]

@LoyVanBeek if you're up for doing all that work, good on you, and I may help out a bit. However, I agree with @kyordhel

I think writing the rule book this way reinforces a problem that I believe (and others too) that we should start solving: as of now, the robots are just following a script; there is no actual push for high-level problem solving. To be fair, some Stage II tests are going in that direction, but their complexity still relies on following a script.

I'm fine doing this for Stage I tests/new teams: go there, do this, come back, etc.

But I'm getting the feeling that veteran teams are going to get bored, if they aren't already. Once a team has the basic functionalities down (navigation, vision, manipulation), all they're really doing to prepare for this competition is build a big state-space machine. We end up awarding points to those teams whose developers themselves have considered all the possibilities in their state-space machines, instead of awarding points to the teams that have a robot that corrects itself and expects the unexpected.

[LoyVanBeek]

Allright, I'm totally fine with rewriting only Stage 1 challenges as GPSRs and pushing the envelope more in Stage 2. Some categories of EE-GPSR are still very difficult with no attempts made on them AFAIK.

As a suggestion for Stage-2: I just want a robot do fully do my laundry :-)

[kyordhel]

Incompatible with new schema