Show room test

[leanderb]

• new type of test which shall be scheduled before all other tests
• the idea is, that no team has to leave the competition with 0 points and the teams can show in a well known setting what their robot is able to do
• each team defines the task for their self with the following rules: o at least 2 tables shall be used; the tables shall be selected from the existing arena o 2 different objects shall be manipulated o start position and end position of the robot is defined by the team o objects are placed on the tables by the teams
• the scoring: o manipulation: 50% of the points as defined for other tests o navigation: 100% of the points as defined for other tests
• depending on the selected task, the maximum number of points is different and this is intended; see the following examples: o a new team wants to show that the robot can pick and place on a 10cm table, the team selects 2 object (of course the ones that works best); the robot picks all objects from one table and place these objects on another table; the spectators cheer enthusiastically and the team gets 250 points (2x100 for picking and 2x 100 for placing multiplied with 0.5; 2x25 for reaching the service area) o a very good team wants to show off and decides to pick from RTT, place on shelf, pick from RTT and place on precision placement table; the spectators are impressed and the team gets 475 points ( (200+200+150+150)0.5 + 325)
• to relax the competition schedule, a test had to be removed (i would prefer a BTT and not the BMT, because the BMT is still important for new teams)
• each team have a 15min timeslot and can restart as many times as they want; a restart is a complete reset of the arena and robot
• it is not allowed to change the self-defined task

[marcomasa]

I think this leaves the TC and OC with too much overhead. It also "destroys" comparability between teams. And tbh, i think that the teams that got 0 points during the whole competition recently would also not be able to perform such a staged setup.

Regarding the exchange of tests:

BTTs are used to introduce more difficult elements step by step, removing one would make the assignment of these elements even more difficult.

I think that the required robot performance of the league is quite increased by our 2 "special" tests (RTT and PPT) which also have not seen a lot of (successful) engagement in recent years. I guess if we actually want to stretch our schedule and/or make the league a bit easier, these two robot capabilities are the first to drop. RTT manufacturing onsite has been quite stressful for years now anyway.

We could discuss this in our roadmap talk.

[woutzen]

I think a show room test is a good idea and i don't expect the overhead to be that much.

The teams could be responsible for arena setup themselves, e.g. by adding another 5 minutes between two 15 minutes runs for parallel ref discussion of the last run and arena setup by the next team.

I would suggest not to use the refbox, but let the teams implement the tasks in their own testing setup. This would reduce the bottleneck "refbox compatibility" for new teams and allow them to leave with more than 0 points, especially if the usage of bags is banned. By this, no additional overhead for task generation would occur for the TCs.

The only overhead is with the OCs who would have to write individual ref sheets for 2 objects and up to 4 workstations for up to 10 teams. This could be reduced by either providing a ref sheet template and letting the teams fill them (this could also be seen as the official task definition by the teams) or by expecting a certain syntactical form of the teams' task definitions from which the ref sheet could be auto generated.

I would schedule the show room test before the final. More experienced teams usually use the time during the cup for preparation of e.g. RTT, PPT. The newer teams could use this time to prepare and test their individual task. Besides, this will probably be a task where most teams will be successful, so that it is interesting to watch for visitors.

The task definitions should be submitted by the teams to the OCs at a specific time, e.g. 24 hours before the run.

The usage of arbitrary surfaces should be forbidden, because otherwise teams could use a white paper as arbitrary surface for more points. There is no clear line, what AS really is an advanced level of difficulty.

Regarding the exchange of tests:

The BTTs could be rearranged to two instead of three. Relevant parts of the BTT are:

• 10cm tables only
• Barriertape
• AS
• Different table heights
• physical obstacles (blocking/semi blocking/none blocking)
• shelf
• containers

These could be rearranged into BTT1:

• 10cm tables only
• Barriertape (one blocking, one semi blocking, one none blocking)
• AS
• Containers

BTT2:

• Different table heights
• Shelf
• AS
• Physical obstacles (one blocking, one semi blocking, one none blocking)

I think, RTT and PPT are important tests for the more experienced teams as an additional challenge, but maybe they could be combined to reduce the organisational overhead. Not in the way of picking from RTT and placing in PPT, but maybe picking 2 objects from RTT, placing them on a third workstation, picking 2 objects from that workstation and placing them on PPT.

[marcomasa]

I will try to respond to the main keypoints:

Making it easier for new teams

I think we have to define the term "new team". Most robocup participants are not hardcore veterans, so they attend to learn and exchange. Teams with a history in the league mostly only have their name kept, the members are often completely replaced when the oldies finish their studies and leave the university.

Of course the new members could copy the code base, but the fun part is developing the robot and understanding the solutions for autonomous behavior. The bonus for robocuppers is that we can compete against each other and therefore benchmark our systems, which is nicer when you know why things work out well.

It could also be that the new team just doesn't understand the old system due to poor documentation and therefore also has to start from scratch.

I think it's a good approach to make things easier for such constellations to participate in our league with success. However, I don't think this can be done effectively by adding a showroom test.

Official run

Our league consists of standardized benchmark tests that every participating team has to perform. If everyone does something unique, we cannot compare teams and therefore should not include such a test in the official scoring.

Schedule before the final

I would not want to sacrifice the testing time for the final benchmark. For good teams, this is sometimes the most valuable time of the competition and can decide a title.

Refbox Bottleneck

We could upload example bag files for all test types. This way, teams can test (limited) stuff without the need for a working refbox setup at home.

Parsing these basic instructions should not be a big challenge when a team is programming an autonomous robot, at least in my opinion.

Task Difficulty

I have not seen a lot of perfect runs from anyone lately. We should encourage the good teams to work on reliability rather than niche cases. See also Roadmap and updated benchmark scenarios for that.

Roadmap

Our league is called industrial@work. I think we struggled to find an identity that meets that name in recent years.

Key aspects of industrial applications are speed and reliability, which should get back into focus a bit more. Higher bonuses for perfect runs and remaining time could achieve that.

Updated Benchmark Scenarios

I am proposing updated benchmark scenarios that should allow new teams to keep in the competition for longer and shift the focus to reliability. Good teams have to consistently perform good and fast to beat their opponents.

BMT - Level 1

• 3 Objects = No need to approach workstations twice
• limited selection of objects (easy ones) --> Teams can't select but can focus on more simple stuff

BTT1 - Level 2

• 4 Objects --> introduces inventory management
• 3 tables that are not neighbors --> introduces navigation
• physical obstacles --> laser sufficient as obstacle avoidance
• No AS --> easier vision for another test

BTT2 - Level 3

• 5 Objects --> introduces task optimizing
• 4 tables --> introduces different perspectives
• visual obstacles / barriertape --> introduces need for visual obstacle avoidance

BTT3 - Level 4

• 6 Objects
• 4 tables
• 2 placements on a shelf --> introduces shelf
• 2 AS --> introduces more difficult perception

BTT4 - Level 5

• 7 Objects
• 4 tables
• 2 picks from 2 different shelfs --> introduces lower shelf picking
• 2 placements into containers --> introduces target objects

Final - Level 6

• As before without RTT and PPT

Increasing Difficulty / Challenges

Precise Manipulation

PPT was also depending on luck and not mismatching the holes. I'd rather see teams pick from a container because it's probably more relevant to actual co-factories. We could start off with only a single object inside of a container.

Moving Objects

RTT concept with decoys seems weird to me as a real world scenario. Maybe bring back traditional conveyor belts? They even used to only start when a robot reaches the position.

Technical Challenges

Keep the Human Coworker Challenge

I like the idea of having the robot and a human co-operate. It also requires teams to detect assembled products and differentiate them from single components. Has relevance to real world scenarios for me and demonstrates possible future scenarios / applications to viewers

Tbd

[steup]

I will only comment on the show room test because the generic change of runs is something different and should be discussed separately.

I think the show room test is a very good possibility for new teams to gain motivation to keep participating in the league. This is simply because it removes the randomness from the tasks. The overhead for the OC is minimal, since we can shift the specific OC work, like preparing Ref-Sheets and preparing bags, to the teams. This also creates experience in the new teams regarding the league's infrastructure and makes the team more integrated into the league's internals. The task is planned to be very limited in points, so the effect on the comparison of established teams is negligible. The time effort for established teams is also minimal, as they can simply run a random task and be done.

I see no major drawbacks in this run. Of course, because of limited time, other runs needs to be culled or modified to fit the schedule.

[steup]

Regarding the change of tasks

I suggest fusing PPT and RTT with BTT2 and BTT3 to create more interesting and versatile runs. I would also rename to Advanced Transportation Task to indicate this. BMT, BTT and SRT (Show Room Task) would become the new beginner tasks and ATT1, ATT2 and Final will be the new advanced tasks. I would remove limitations on combination in the upcoming years like: Allowing containers in shelves and on RTT. Put Arbitrary surfaces on PPT and RTT. Transport from RTT to PPT. This also leaves us with the same number of tasks as before; However, we may need to change the times for the individual tasks.

[leanderb]

show room test not implemented; see #64