robocup-at-work / rulebook

The Rulebook for the RoboCup@Work league.
8 stars 8 forks source link

Role of Team Referees and Technical Committee during Competitions #61

Closed marcomasa closed 1 year ago

marcomasa commented 1 year ago

In the past, we've always requested that teams assign one member to watch all team competitions as a referee. The tasks of the refs were to count points and discuss point deductions and other penalties.

In RC 2022 world cup final, we had a rather unpleasant situation that none of the refs noticed a title-deciding collision. It was only noticed when a TC member (myself) called it out, after which the 'stop' call was made by the performing team.

Afterwards the discussion came up whether or not the penalty would have to be applied or not, since none of the refs made the call during the Run Phase.

This leaves us with a couple of critical points:

1. Should we change the role of Refs

Most teams use new team members as their ref as they "don't need their working power" so much during the competition (harsh wording, I know, but let's be real here). These new members are often not very familiar with the rule book and therefore are (in my eyes) not suited for such critical decisions. We either have to do briefings for the teams / refs before competitions begin, or we change the role of refs. If we say that refs are more of a "double check" mechanism that ensures that someone of every team counts their points, it would also encourage the refs to pay a bit more attention. However, it also required us to think about the second point:

2. Should we give the TC more responsibility during runs?

As the TC writes the rule book, they are more likely to know the rules better. They could be suited better for decisions regarding points and penalties.

They are, however, mostly team members as well and therefore might favor some teams. So we'd have to still have some refs that double check.

If more than one TC member is available onsite, we'd also have to instate a "Head of TC" or something similar that has a deciding vote-value in case of a tie (e.g. 1.5 instead of 1).

3. Ensuring that refs can do their work

I've also received some complaints of refs that they were not able to move around the arena as much as they wanted and/or needed to during runs of teams. The current arena setups don't allow spectators to see everything in detail from every angle, as walls often block the view partly, so refs have to move quite a lot during a run.

We've always had some free space between the arena and the public spectator area. We might want to only allow refs / TC to be inside of that "internal" spectators area. All other members could be asked to view from outside.

Implementing #60 could also help, but that's a lot of additional work for the committee, which is already a bit overloaded already.

woutzen commented 1 year ago

1.+2. Role of TC and Ref:

The reason to differentiate between TCs and Refs is that TCs as legislative decide about the rules in advance and the Refs as judiciary evaluate during competition if the rules are met. The Refs are supposed to follow the rulebook exactly. That doesn't require experience in the league, but it does require extensive knowledge of the current rulebook.

If we would mix both roles and give the TCs the chance to change/reinterprete/add rules after an undefined situation occured to decide about the rating of that situation, we don't really need a rulebook at all. The teams could not rely on the predefined rules and could not prepare accordingly. If the TCs would be the ones judging a run, they would decide according to their understanding of the rulebook as intended during rulebook writing and not according to the exact words used. This would be unfair for new teams that use the rulebook as a basis for their development.

Therefore it is very important to keep the differentiation between TCs and Refs to ensure fairness, and even follow it more strictly in the future!

There are two different cases:

  1. Ref decisions during run time

Briefing of the refs before the competitions is a good idea. The refs need to have a very good knowledge of the rules. Every team can decide, which team member should be the ref. Therefore, they have the freedom to choose someone that has more experience in the league and they can brief and train their ref in advance (including making sure that the ref actually reads the rulebook). If their ref is not qualified and able to make the required decisions, it is the teams fault and the currently competing team should not be punished by that. Therefore, if the Refs don't call an obvious major collision it can not be argued to be a major collision afterwards.

  1. Ref decisions after run time

After the run all refs should gather in an area, no one else has access to (no TCs, ECs, Team Leaders)! They discuss their observations and decide according to the written rules about the rating. I would suggest that if something unforeseen happens in a run, that is not clearly defined in the rulebook, the result is always favoring the current team. There is no decision about a rule necessary to do so, therefore the refs are capable to to that. For all following team runs of the current task that situation is solved similarly. This also keeps the discussions between runs of a single task small.

After the task, before the next task, the TCs can discuss the situation and decide on new rules or handling of undefined situations. These decisions only have an influence on future runs, not on past ones to solve similar situations.

Separation of roles

Refs are supposed to be neutral. Obviously, they are team members as well, which makes them slightly biased (thats why one Ref from every team is so important), but it should be their goal to decide neutral and fair.

Team Chefs are not at all supposed to be neutral, they are supposed to represent the interests of their team. Since in our league TCs and Team Chefs are often the same person, TCs can not be neutral judges. If TCs would decide and Refs would double check the results, assuming the refs are unexperienced team members, they won't question any TC decisions, but just nod and agree.

A head of TC similar to a Head ref that has the power to influence any decisions can never work in a fair way. The head ref is a passive, organisational role on purpose. He/she is only supposed to lead and maybe counsil but never decide. A head of TC would have way too much power and would favor his own team. In case of a tie in a ref decision, it should be decided in favor of the currently competing team. In case of a tie in Rule discussions between tasks for future tasks, the EC is the tie-breaker.

I suggest a rule that forbids a TC or a Teamleader to be ref in the same time.

3.Ensuring that Refs can do their work

Walking around the arena to see everything is actually always a struggle as a ref. Therefore it is a good idea to block the space for everyone apart from refs. The TCs dont need to be in that area either in my scenario. This area could also be used as a space for the refs to discuss the ratings after the run without disturbance by anyone.

Also, the refs should be encouraged to spread out within that area, as it was done a few years ago. Not every ref needs to see every situation in detail and having refs at more or less fixed positions around the arena is enough, as long as at least two refs can see what is happening.

marcomasa commented 1 year ago

First of all, thank you for your detailed feedback!

Regarding some of your points, I think it might be a matter of our roadmap for the league how we actually decide. Nonetheless, I will try to reply to your suggestions as I understood them:

Ref Briefing

Briefing the Refs before the competitions begin should always be on the agenda. The rulebook is quite long and things might be missed while reading it. Also, some phrasing could be clear to the TC but not to a new reader due to the missing background info. I suggest we even add it to the proposed schedule table in the rulebook.

TC Judging

I do agree that giving the TC too much power might make things unfair, so we can keep the current system of one ref per team.

Ref "Shortcomings"

The rules that the TC makes are there to guideline the competition. It takes a lot of effort to think about and discuss these rules, so I dislike the idea that teams can "get away" with breaches just because the refs miss out on a situation. Obviously this can be for various (unintentional) reasons, e.g., not fully understanding a rule, not paying attention, bad view, new to league, busy documenting a previous event, etc... The possibility for that to happen probably gets smaller the more teams (and therefore refs) participate, but it could still be a problem for robocups with only a few teams attending.

I think this is one of the points for the roadmap / league philosophy. To me, each Robot should always follow the rules as they are implemented to also act as safety measurements. So it should not be a matter of who actually noticed (major) rule breaks.

If we want more fairness, we should try to decrease the room for human error. I'd like to (officially) allow TC members to call (major) breaches to have some sort of backup for such a situation. I don't think we specifically excluded them from doing so in the current rule book, but as it came up for discussion in Bangkok, let's add it to make it clear. We could limit the call power (collisions) to ensure they are not involved too much.

Head of TC

The goal of the committee should always be to keep the league's philosophy and intention of rules. While there might be different opinions on how this looks in reality, the TC members have always come to a mutual agreement in the past after exchanging their views on a matter. For ties, I don't see much of a difference if it's a Head of TC or EC decision, as both are (former) members of a team. Head of TC could be more into the matter / discussions, but that can be both positive and negative, depending on the actual case.

New aspects

I think you also brought up some new aspects that i will try to adress here:

Teamleader/TC != Ref

TC part is discussed above, but I would not forbid a committee member from doing additional tasks if they can. TL part is fine in any way for me. I think every team member will probably favor their own team to some extent, which is fine as long as we have control instances in form of other refs (and backup).

In the end, we still have a section in the rulebook about sportsmanship that everyone should follow. Teams that openly try to cheat will just get banned, but gladly we did not encounter such a situation.

Postponing decisions

I think it might help to discuss while the memory is fresh. Some things might also happen to following teams as well, and the decision could be further influenced by that.

Always in favor of the current team

Difficult to generalize, I think. It is hard to really fix all shortcomings in the rulebook beforehand, as some are only noticed when a specific situation brings them up. We would kinda allow things that are not specifically forbidden, which could pave the way for some ugly moments. Yet again, I am thinking about safety issues. But also with point discussions, favoring can mean multiple things (giving full points, not giving penalties, etc.)

I think we handled things quite well all together in BKK and found solutions that did not penalize too much but kept the spirit of the existing rules. I also think that every team that attended did profit once or more from such decisions, so to me, it felt okay to do so.

Refs gathering in private

I do agree that they should be able to do their work without being disrupted by non-refs. I do not recall anyone actively interfering, but that may just be my perspective.

We noticed that some rules were not defined clearly enough when refs kept coming to ask the TC about how to interpret rules. To me, it makes sense to not "lock the refs in a room" so they can still do that. Yet again, writing a watertight rulebook for every situation is a challenge and I don't think that we can do that for every possible future scenario, so I guess this will happen again.

steup commented 1 year ago

General Strategy for unclear rules on the run

The general problem is that there will never be rules which are exact enough to not leave room for interpretation. Consequently, we need a strategy to handle situation, where the rules are not clear. I favor simple generic strategies to cover such situations like always in favor or always against a team because these do not generate discussions afterward.

Separation of TC and Refs

In my opinion, a separation of TC and Refs is meaningful to guarantee a fair interpretation of the rules. Refs resemble judges, whereas the TC is like a parliament. If we mix these two, it is easy to “change” rules during the runs by changing the interpretation on the fly. To prevent this and guarantee a similar assessment for all teams, a separation is necessary.

Independence of Refs

I think for a fair analysis of runs and concrete situations during the runs, the refs require space to discuss this among them. Consequently, I am in favor of shielding them from the teams to prevent outside influence. I think we also need a strategy to handle “proofs” like videos. I suggest we either go with the “first decision is final strategy” or we allow discussion afterward. But the discussion needs to be guided and controlled by the refs. I think for major collisions we require special rules as these do not allow for discussions. Additionally, we need a way to handle outside influence during a run regarding major collision (e.g.,: someone not being a ref stating a major collision)

leanderb commented 1 year ago

TC meeting: