Restructuring of the Benchmark Tests

[marcomasa]

We are proposing updated benchmark scenarios with the intention of fulfilling the philosophy of the league as described in #62 .

The exact details are yet to be discussed by the TC, but we've agreed that we want to make the overall competition a bit easier, take away some special cases and shift the focus on reliability and speed.

We therefore want to eliminate RTT and PPT from the regular competition and also introduce the more difficult aspects of our league a bit later. RTT and PPT might become Technical Challenges, so Teams that put effort into those can still show something.

Note: RTT also is a lot of organizational effort in international competitions, so we were discussing if we fully discard the test. But the aspect of manipulation of moving objects is also interesting, so there is no final decision yet. What we agreed on is that we should also allow simplified objects (see #63 ) as the focus is the movement, not (just) detection.

This achieves:

• New teams can do more of the official tasks with a lower intro boundary.
• Good teams have to consistently perform reliable and fast to beat their opponents.
• One test less which releases schedule tightness onsite
• (No need for an RTT)

Basic Manipulation Test - Level 1

• 3 Objects = No need to approach workstations twice
• limited selection of objects (easy ones) --> Teams can't select but can focus on more simple stuff

Basic Transportation Test 1 - Level 2

• 4 Objects --> introduces inventory management
• 3 tables that are not neighbors --> introduces navigation
• physical obstacles --> laser sufficient as obstacle avoidance
• No Arbitrary Surface --> easier vision for another test

BTT2 - Level 3

• 5 Objects --> introduces task optimizing
• 4 tables and varying heights --> introduces different perspectives
• visual obstacles / barriertape --> introduces need for visual obstacle avoidance

Advanced Transportation Test 1 - Level 4

• 6 Objects
• 4 tables with varying heights
• 2 placements on a shelf --> introduces shelf
• 2 AS --> introduces more difficult perception

ATT2 - Level 5

• 7 Objects
• 4 tables with varying heights
• 2 picks from 2 different shelfs --> introduces lower shelf picking
• 2 placements into containers --> introduces target objects

Final - Level 6

• As before without RTT and PPT

[leanderb]

I do not like the idea of removing RTT and PPT. Especially PPT is very nice for spectators. PPT will become easier, when introducing QR tagged objects as discussed in #63.

I agree, that a separate RTT and PPT test requires a lot of time. So we can just move it to BTTs or ATTs.

Both RTT and PPT are former technical challenges. The idea was to introduce this tasks and keep them separated until several teams participates successfully. So before we degrade this tasks, let's discuss the future role of technical challenges.

[marcomasa]

Discussed solutions by TC:

very pretty increase of difficulty achieved

Basic Manipulation Test - Level 1

• 3 Objects = No need to approach workstations twice, no need for intense planning

• limited selection of objects (easy ones) --> Teams can't select but can focus on more simple stuff

• Object types are preselected. Suggestion: small grey alu (f20_20_g), small nut (m20), bolt (m20x100) (Object selection also prepares for co-worker assembly test technical challenge)

Basic Transportation Test 1 - Level 2

• 4 Objects --> introduces inventory management
• 3 tables that are not neighbors --> introduces navigation
• All object types (not preselected ones)
• No Arbitrary Surface --> easier vision for another test & constant table heights

BTT2 - Level 3

• 5 Objects --> introduces task optimizing
• 2 Decoy Objects --> better vision necessary
• 4 tables and varying heights --> introduces different perspectives
• 2 physical obstacles --> laser sufficient as obstacle avoidance

Advanced Transportation Test 1 - Level 4

• 6 Objects
• 3 Decoys
• 4 tables with varying heights (+ 1 shelf) = 5 Service Areas
• 2 visual obstacles / barriertape --> introduces need for visual obstacle avoidance
• 2 placements on a shelf --> introduces shelf
• 2 AS --> introduces more difficult perception & table height is not constant anymore
• 1 placement in PPT

ATT2 - Level 5

• 7 Objects
• 5 Decoys
• 4 tables with varying heights (+ 2 shelf) = 6 service areas
• 3 Arbitrary Surfaces
• 1 pick from 1 shelf --> introduces lower shelf picking
• 2 placements into containers --> introduces target objects
• 1 placement in PPT
• 1 visual and 1 physical obstacle

Final - Level 6

• 8 Active Service Areas
• 4 Arbitrary Surfaces
• 2 visual and 2 physical obstacle
• 10 Objects (Introduces heavy planning requirements)
• 8 Decoys
• 2 visual and 2 physical obstacle
• 2 PPT Placements
• 2 picks from 2 different shelfs
• 1 place on shelf

Resulting Task Table

[steup]

As discussed in the extended TC Meeting on 08.01.2023: The following comments provide ideas to handle the RTT in the upcoming and future competitions, to be votes on by the extended TC (TC+EC)

[steup]

RTT Integration Option 1: The RTT will be done now and in the future as a technical challenge until there is sufficiently interest in competing teams, at which point it may be reintegrated into the normal competition

[steup]

RTT Integration Option 2: The RTT will be done in technical challenge to develop and test (optional) simplifications and their point allocation for the 2023 seasons. In the season 2024 it will be reintegrated into the ATT Tasks.

[steup]

RTT Integration Option 3: The RTT will be extended with optional simplifications and will be integrated into at least one ATT this season (2023).

[steup]

The following Options are possible simplifications to make the RTT more accessible to beginner teams:

[steup]

RTT Simplifcations Option 1: Add some static QR-Tag markers to the table to ease speed detection and rotation direction detection.

[steup]

RTT Simplifications Options 2: Replace objects on the RTT with boxes with QR-Tags to ease vision related issues to detect objects and decoys

[steup]

RTT Simplifcations Options 3: Extend distances of objects and decoys to ease detection and grasping

[steup]

RTT Simplifcations Options 4: Remove 1 or more decoys from the task

[steup]

RTT Simplifications Options 5: Fixate speed and turning direction of the RTT

[steup]

All the options should have an appropriate point deduction and may or may not be combinable. I ask everyone to vote on the options using the up and down vote emoji. TC and EC votes will be counted.