New Scoring for Follow the Path

How Things Are

Follow the Path has a time-based scoring method with time penalties for hitting obstacles. Going through gates out of order or in the wrong direction automatically terminates the run and assigns the maximum time. This scoring model has some consequences we'd like to avoid:

It seems to place too much emphasis on speed.
It is less forgiving than scoring similar tasks in the physical competitions, which means all teams that miss a gate get the same score.
If we relax the rules to allow teams to continue even if they skip a gate, the time-based scoring gets complicated.

Proposal

To address the problems above, we could transition from time-based scoring to point-based scoring, in which points are awarded for passing through gates and deducted for hitting obstacles and making mistakes. At a high level, the scoring method should reward teams that display evidence of intentional integration of control and perception to navigate the channel. In particular, teams should be rewarded for:

Entering the course through the start gate,
Correctly navigating through gates (teams that pass through more gates should generally do better than teams who pass through fewer gates),
traversing gates in the correct order and direction,
staying on course,
and avoiding obstacles. Among teams that do the above equally well, time can be used as a tie-breaker.

The following scoring system is proposed to achieve the goals above:

Teams must pass through the start gate first to score points.
1 point is awarded for each gate traversed in order.
If a team skips a gate, it can still get points for traversing later gates. However, once a gate is traversed, all previous gates are out of play (so teams can't get points by going back).
Penalties are deducted for each collision and for navigating through gates in the wrong direction.
Passing through both the start gate and the finish gate ends the run and stops the clock.
In the event of a tie, the team with the lowest time wins.

Staying on course

Two alternatives have also been suggested to reward "staying on course":

Longest Path Method

A bonus is awarded equal to the highest number of consecutive gates traversed correctly.

Consecutive Pair Method

Two points are awarded for the first gate.
A bonus point is awarded for each gate traversed if the immediately preceding gate was also traversed.

Comparison

In both methods, the maximum score is twice the number of gates.
In the longest-path method a team that misses a gate in the middle does worse than a team that misses a gate at the beginning. This seems weird.
The consecutive pair method allows a run that traverses the first 4 gates in a row and then drives off course to score the same (8 points) as a run in which the USV goes through gates 1-2, 4-5, and 7 (i.e. skipping 3 and 6 then going off course). This seems less weird.
Based on the above I'm in favor the consecutive pair method.

Normalizing Scores

To prevent difficult runs from being weighted dramatically higher than easy runs with fewer gates, the total score for each run could be divided by the number of gates in that run.
This will have the effect of making it possible for a team that does well on difficult runs but makes a catastrophic mistake on easy runs tie with teams who do well on easy runs and fail at harder ones. In other words, the return on effort reduces as the runs get harder. I'm not sure this is desirable.

I like this - the description is clear and should be easy for the teams to follow. A few thoughts:

Clarification needed - if a team passes through a different gate before the start gate, are they unable to earn any points? Or does the course "start" once they do pass the start gate?
Is it difficult to manually check the time in case of a tie? We could replace this with a proportional time bonus that has a maximum value approximately equal to the point value of passing a single gate. This would spread the scores out a bit, but wouldn't make skipping gates to rush to the end a profitable strategy.
Given that a skipped gate is out of play, we are already rewarding staying on course. For simplicity and ease of explanation, maybe we should just leave it at that (at least for the first round with the new rules).
We could balance between number of gates and difficulty of the courses by using a weighted normalization for the scores: divide by the number of gates, but then multiply by a "difficulty factor", so that rather than a 9-gate course having triple the possible points of a 3-gate course, we could set it to 1.5 times - or whatever seems logical for a value.
Let's make each gate worth 10 points so that we don't end up with small decimal math :) 5 point penalty for a collision or violating the rules of the road?

Clarification needed - if a team passes through a different gate before the start gate, are they unable to earn any points? Or does the course "start" once they do pass the start gate?

Good catch. I think the course "starts" once they pass the start gate.

Is it difficult to manually check the time in case of a tie? We could replace this with a proportional time bonus that has a maximum value approximately equal to the point value of passing a single gate. This would spread the scores out a bit, but wouldn't make skipping gates to rush to the end a profitable strategy.

Can you explain what you mean by "manually check the time"? Since I don't understand yet I'm not sure what you're proposing to replace.

Given that a skipped gate is out of play, we are already rewarding staying on course. For simplicity and ease of explanation, maybe we should just leave it at that (at least for the first round with the new rules).

Yeah, I agree. Maybe the problem is "staying on course" doesn't quite capture the goal. I think we wanted a team that passes through, say, 3 consecutive gates to do better than a team that passes through 3 gates that are not consecutive. The reason might be better described as "demonstrating intent." If it passes through consecutive gates it looks more like it's trying to navigate the channel.

We could balance between number of gates and difficulty of the courses by using a weighted normalization for the scores: divide by the number of gates, but then multiply by a "difficulty factor", so that rather than a 9-gate course having triple the possible points of a 3-gate course, we could set it to 1.5 times - or whatever seems logical for a value.

We could, but then we have to have a criteria for assigning "difficulty factor" and there is a cost to score complexity too, so I'd prefer to avoid this if possible. I think the difference in scores between a 3-gate course and a 9-gate course is not insuperably huge. I guess an equivalent way to put this is that I think the "difficulty factor" can just be the number of gates.

Let's make each gate worth 10 points so that we don't end up with small decimal math :) 5 point penalty for a collision or violating the rules of the road?

Yeah, good idea. I'm not sure what the collision penalty should be. Maybe 3 points? That way it's almost always better to try to get another gate than to cut your losses and run, and teams that pass through more gates are likely to beat teams that pass through fewer gates.

Similar to the clarification about the start gate, I think we need to clarify what happens when a gate is crossed in general. Does it stop being part of the scoring? Or is it still possible to lose points by going backward through a gate that's already crossed correctly?

In the interest of realism, I think it's probably better for it to remain active. However, this would mean it's possible to get blown backwards through a gate you just crossed. There probably needs to be a threshold that prevents this (we might have one already).

We also discussed the possibility of terminating the simulation if a gate is crossed in the wrong direction on the basis that this indicates a major violation of the rules of the road. However, since it's possible for environmental factors to push a vehicle back through a gate even when it's "intending" to go in the right direction, I think it's probably better to apply a penalty, unless we have a neat way of distinguishing between a vehicle that got pushed and one that's outright driving in the wrong direction. Maybe 5 points would be reasonable?

osrf / vrx