Proposal: Require a second scramble checker for possible WRs

[lgarron]

It seems that we can't fix the mis-scrambling issue in general, but it keeps happening for high-profile cubers who are obviously fairly likely to set a WR (cf. Jeff Park).

I'd like to propose the following limited scramble checker policy:

---

If a competitor has an official PR single better than the WR average, then every scramble for the competitor in that event must be checked by a second scrambler who:

1) is a different person than the one who scrambled the puzzle, 2) must verify that the scramble is correct for the current attempt, and 3) sign the score card to certify this.

Every incorrect scramble must be fixed and reported to the Delegate.

---

Ideally, I'd like to modify score card software to automatically and clearly indicate such competitors, with a column for the scramble checker to sign.

And if this turns out to be successful and practical, we can think about expanding the scope – e.g. applying this for CR or NR candidates, applying this to top competitors from previous rounds, or allowing other competitors to specifically request this under certain conditions.

@thewca/wrc-team: would you be in favor of this proposal if I worked on implementing it?

We can also try this out without immediately integrating this into the Regulations, but I'm getting really frustrated with preventable incidents, and want to get consensus that it's worth doing something like this.

[kingmathyall]

I'd be in favor for it, but I'd like organizers/delegates to also be able to specifically request this rather than just the competitor.

One reason for this is because competitors such as Graham Siggins, who is clearly capable of getting a WR for 5BLD or possibly even MBLD, however he has no singles below the WR and doesn't currently have a WR.

It might be a good idea to allow this feature to automatically be applied to competitors who are in the top 10 or 20 of whatever that certain event is - should cover the majority of people capable of having getting a WR in that event.

[xsrvmy]

One valid question is: does this apply to 6x6, 7x7 and megaminx? Also, what if the two scramblers disagree on what the lightest/darkest colors are if, for instance, black is used instead of white?

[lgarron]

One valid question is: does this apply to 6x6, 7x7 and megaminx?

Yes, at the discretion of the Delegate.

Also, what if the two scramblers disagree on what the lightest/darkest colors are if, for instance, black is used instead of white?

Then they should talk and come to an agreement. I don't recall ever encountering a puzzle without a deterministic decision. (If the scrambler treats black as white, then they are simply scrambling incorrectly. See 4d+.)

[xsrvmy]

Then they should talk and come to an agreement. I don't recall ever encountering a puzzle without a deterministic decision.

I was thinking about megaminx but I guess delegate can intervene if that happens.

Talking about megaminx, the regulations does not make clear what orientation to use if there are two different shades of green next to white. Presumably the darker one?

[lgarron]

Talking about megaminx, the regulations does not make clear what orientation to use if there are two different shades of green next to white. Presumably the darker one?

Good point – in fact, this can apply to any puzzle. It's probably worth cleaning up the language or adding Guideline. Could you file a separate issue (and even a PR, if you feel like it)?

[lgarron]

One concern I've heard people bring up is that this proposal singles out fast competitors as more worthy of attention and competition resources. I have two responses to this:

1) Mis-scrambles probably affect many competitors, but high-profile competitors are currently suffering under unfair conditions because it's usually their best solves that are scrutinized and invalidated.† This proposal tackles that unfairness. 2) I'd love to institute a system that improves scramble accuracy for all competitors, but I have historically not found enough support from other people to tackle the problem at full scale. However, this proposal can easily be extended to all competitors in the future.

† There certainly never seem to be repercussions for organizers, Delegates or scramblers – even with patterns of repeated incidents.

[lgarron]

If a competitor has an official PR single better than the WR average

I got some stats (from Judith Laister) on how many competitors this includes in each event right now:

• 222: 451
• 333: 65
• 444: 4
• 555: 4
• 333bf: 12
• 333oh: 51
• 333ft: 2
• clock: 12
• pyram: 123
• skewb: 92
• sq1: 21
• (666: 1)
• (777: 1)
• (minx: 2)
• (333fm: 74)
• (444bld: 2)
• (555bld: 1)

451 competitors for 2x2x2 might seem a bit high. On the one hand, most of those competitors are likely to set a very fast time on a sufficiently easy scramble – the prevalence of so many fast solves validates the need to take extra care in avoiding easy mis-scrambles. But we can also adjust the proposal to include fewer competitors if others feel this is too many competitors right now.

[HippolyteM]

Are you all suggesting to treat the competitors / apply regulations differently according to their speed? Erk. Maybe you should try something else than speedcubing.

[lgarron]

Are you all suggesting to treat the competitors / apply regulations differently according to their speed?

Yes. Please read the comments before yours.

Also, please focus on constructive ideas and avoid insulting anyone in these discussions.

[HippolyteM]

Ok. Please excuse me my comment, which might have been too agressive.

I just wanted to tell that it makes me sad to see such proposals, which are soooo far away from the WCA spirit, and which goes the exact opposite way as our regulations usually do.

I'm not against scramble checkers if we find a good way of doing it. I'm definitely against having fairer conditions for some.

[kingmathyall]

Don’t forget about the following regulation! https://www.worldcubeassociation.org/regulations/#A2d1

Kind Regards, Matthew Dickman

Sent from my iPhone

On Jul 29, 2018, at 3:10 PM, Hippolyte Moreau notifications@github.com wrote:

Ok. Please excuse me my comment, which might have been too agressive.

I just wanted to tell that it makes me sad to see such proposals, which are soooo far away from the WCA spirit, and which goes the exact opposite way as our regulations usually do.

I'm not against scramble checkers if we find a good way of doing it. I'm definitely against having fairer conditions for some.

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Laura-O]

I agree with @HippolyteM in some kind of way, but the reason why I think checking only fast competitors is bad is the organizational overhead and might cause side-effects and/or other incidents. For events which take more time to scramble, the puzzle has to be kept until one of the other scramblers is available, which might be difficult on a busy scrambling table. For quicker events and especially when using the popular approach of leaving the scoresheet in the box, I am pretty sure scramblers will tend to forget to check for WR candidates.

As Matthew said, defining potential WR candidates is also difficult. Especially for the events where scrambles matter, I think the numbers are much higher and the sub-average approach is not very precise. For example, 2014BAUD01 wouldn't be on the list for Skewb, but has average WR rank 9.

However, I agree that we need to take actions. I am sick of the number of misscrambles and especially the number of scramblers who intentionally ignore they are supposed to check scrambles or even think it's funny to setup misscrambles. I think we should start with requiring scramblers to sign/initial on the scoresheet after scrambling. This takes time and slows down scrambling, but it seems like we have to bite the bullet. Additionally, I would support adding a regulation which explicitly mentions the option to disqualify/ban competitors when they intentionally ignore 1e1 and 1f1.

[lgarron]

I think we should start with requiring scramblers to sign/initial on the scoresheet after scrambling.

I'm still concerned that a single scrambler signing for their own scramble is not enough. In particular, it may not be effective against the common cause of duplicate scrambles, where the scrambler is looking at the wrong scramble row the entire time. A second set of eyes contributes a fresh comparison of attempt # vs. scramble #.

Further, a role of dedicated scramble checker doesn't require ability to read notation or perform the scrambles quickly, so it should be easier to find people who can do it. (In particular, this is a good role for parents of competitors who aren't really cubers but are willing to get involved with competitions.)

I've seen the following setups work without major problems:

• 2 scramblers and 1 checker (2x2x2 finals, Worlds 2015)
• 2 checkers, checking each other for important scrambles (US Nationals 2018)
• 2 pairs of 1 scrambler and 1 checker each (US Nationals 2018, some final rounds)

For events which take more time to scramble, the puzzle has to be kept until one of the other scramblers is available, which might be difficult on a busy scrambling table.

It seems that we are converging on "tipped and solved" => ready to scramble, and "upright and scrambled" => ready to bring to competitor. If we want something less rigid than dividing the table into areas, a "tipped and scrambled" cover that someone placed towards your side of the table can indicate "check this one".

In addition:

• Runners and judges can also serve as a guard for puzzles that have not been checked, since they can see the score card before the attempt is started.
• Competitors can look for a scramble checker signature before they start the attempt.

What do you think of a trial period of competitions trying it one of multiple ways (scrambler signing, separate checker signing, scrambler + checker signing), to see how well each system works for the duration of a competition?

For quicker events and especially when using the popular approach of leaving the scoresheet in the box, I am pretty sure scramblers will tend to forget to check for WR candidates.

At Nationals, fast competitors had ★ stars next to their name. It was easy to notice, and I think it's possible to make "CHECKING REQUIRED" fairly conspicuous if we want. Once a column has a scramble signature, it also becomes a visual cue that the remaining attempts should.

However, I'd be happy to enforce checking for all competitors.

especially the number of scramblers who intentionally ignore they are supposed to check scrambles or even think it's funny to setup misscrambles.

Additionally, I would support adding a regulation which explicitly mentions the option to disqualify/ban competitors when they intentionally ignore 1e1 and 1f1.

Do you have particular incidents in mind?

[Laura-O]

Further, a role of dedicated scramble checker doesn't require ability to read notation or perform the scrambles quickly, so it should be easier to find people who can do it. (In particular, this is a good role for parents of competitors who aren't really cubers but are willing to get involved with competitions.)

Yes, I am sure this does work with some parents and if there are only a few cubes to check (i.e. the approach you originally proposed). It doesn't sound like a good solution for checking all groups as this has to happen quite fast. As an experienced cuber, you are able to scan scrambles much faster than someone who compares every single color.

• 2 scramblers and 1 checker (2x2x2 finals, Worlds 2015)
  • 2 checkers, checking each other for important scrambles (US Nationals 2018)
  • 2 pairs of 1 scrambler and 1 checker each (US Nationals 2018, some final rounds)

No offense, but two of your examples (1 and 3) are rounds with fewer competitors than "normal" rounds. By this way of argumentation, I could also say that nobody needs checkers and list several finals from Euros 2016 and 2018 or Worlds 2017.

I think the ideal testing environment would be a regular round of 2x2x2, 3x3x3, Skewb or Pyraminx with 40 competitors per group, 2-3 scramblers + the checking resource we want to test. In these events, scrambling often feels like an assembly line which obviously makes scramblers less attentive.

It seems that we are converging on "tipped and solved" => ready to scramble, and "upright and scrambled" => ready to bring to competitor. If we want something less rigid than dividing the table into areas, a "tipped and scrambled" cover that someone placed towards your side of the table can indicate "check this one".

Not the topic of this thread, but I suspect this approach results in more duplicate scrambles and cube mix-ups. I already had this impression at Worlds 2017 and it happened again at Euro 2018 (my team wanted to switch one day). So this might as well be something we (as the WCA, not the WRC in particular) could work on.

What do you think of a trial period of competitions trying it one of multiple ways (scrambler signing, separate checker signing, scrambler + checker signing), to see how well each system works for the duration of a competition?

I think we have to distinguish between two factors:

• Applicability: This will be clearly visible when testing the multiple ways at competitions. Are there cubes piling up in front of the checker? Does the checker slow down the round significantly? Are there issues with giving scrambled cubes to the checker?
• Effect: This is not measurable at a single competition. I have delegated competitions with 1-2 potential repeated scrambles. If we used a checker at those competitions, I might have concluded the checker helped, but in fact, it was only a coincidence.

So basically, if we want to test these approaches, we need a broad test with competitions of different sizes, in different regions, with different events, etc.

Do you have particular incidents in mind?

• Max 4.xy OH WR on a misscramble, which was setup by a someone
• Several 3-4 move Skewb scrambles at WCCT Reno
• Several occasions of scramblers just putting the cube in a box without checking
• Several occasions of judges not paying attention, using their phones, talking to others, etc.

I am definitely not suggesting to apply this every time a scrambler or judge makes a mistake. Mistakes happen. But I think it can be helpful to refer to this (similar like we often refer to "Every competitor must be available for judging").

[xsrvmy]

Should a Chinese delegate be involved in this discussion? I believe China already asks the scrambler to sign.

[lgarron]

Should a Chinese delegate be involved in this discussion? I believe China already asks the scrambler to sign.

Indeed. I'd love input from @Baiqiang.

No offense, but two of your examples (1 and 3) are rounds with fewer competitors than "normal" rounds.

Indeed. However, these rounds were more fast-paced than most.

As an experienced cuber, you are able to scan scrambles much faster than someone who compares every single color.

We should probably come up with a good rule of thumb for slower scramblers, like checking U and F for cubes. (If we're concerned that this might not catch mis-scrambles, we can try to detect cubes for whom low-distance mis-scrambles result in very similar U/F patterns.)

[lgarron]

Effect: This is not measurable at a single competition. I have delegated competitions with 1-2 potential repeated scrambles. If we used a checker at those competitions, I might have concluded the checker helped, but in fact, it was only a coincidence.

I would love if all competitions had that few issues. :-) But I agree that more data for any given approach is better. Would you prefer something like this?

• Add a single scrambler signature column to all score cards.
• Each scrambler signs for their own scrambles.
• Pick a set of events (e.g. 2x2x2, 3x3x3, Pyraminx, Skewb, BLD, multi BLD) and require scrambler signatures for a month.
• If there are no huge logistical concerns after a month, start requiring scrambler signatures for all score cards.

After half a year, we can evaluate whether there are still enough high-profile incidents to require an independent scramble checker under some conditions.

[Baiqiang]

China has been forcing scrambler to sign for more than 4 years. It helps in the following:

1. If a competitor reported duplicate scramble, we could be easier to figure out the actual situation.
2. We ask scramblers to sign first so he will know which one should be scrambled and won't mess up cubes and scorecards.
3. Signing a scramble makes the scramblers feel the responsibility of checking scramble status.

I asked Chinese delegates' points. We think it's better to focus on the scramblers instead of scramble checkers. If checkers were needed, it's better to check all scrambles instead of only fast cubers'.

For the record, here's a pic of the scorecard we used in China:

[Laura-O]

But I agree that more data for any given approach is better. Would you prefer something like this?

I don't think it's a good idea to force people to do that. There will be a lot of concerns that signing will slow down the whole scrambling process (which it obviously does, but hopefully not significantly). What I would suggest is having some "trial period" in which delegates can volunteer to let scramblers sign and can share their experience with us. I am pretty sure that there would be enough delegates who test this approach and are also willing to give us reasonable feedback.

[Baiqiang]

There will be a lot of concerns that signing will slow down the whole scrambling process (which it obviously does, but hopefully not significantly). It's not significantly at all in my experience.

[coder13]

Does scramble checking imply that scramblers themselves just do the scramble and not check? Would scramblers just do a quick check to see if pieces are in the right place to then hand over to the scramble checker to take a much more detailed look? What's wrong with having the scrambler do the more detailed look?

[lgarron]

I am pretty sure that there would be enough delegates who test this approach and are also willing to give us reasonable feedback.

Two thoughts on this:

1) Calls to try scramble checkers in the past have not met with a lot of success. But maybe now's a different era. 2) I think we need data from competitions that are not proactive and eager to try this out.

But at this point, I guess a few extra weeks of volunteer testing won't exactly kill us.

Does scramble checking imply that scramblers themselves just do the scramble and not check?

The scrambler is still responsible for performing right right scramble, so they should really be checking it. Whether we require a separate signature is another question. (My initial proposal in this issue doesn't.)

[Laura-O]

Two thoughts on this:

1. Calls to try scramble checkers in the past have not met with a lot of success. But maybe now's a different era.

I think that's a question of how you sell it. Maybe it would also help to pick some competitions and ask delegates personally?

2. I think we need data from competitions that are not proactive and eager to try this out.

The problem is, that we have to rely on the feedback delegates give us. Someone who believes that additional scramble checking delays a competition significantly might give us exactly this feedback, no matter why the competition was delayed. So I think we need data from both sides.

[Goosly]

@Laura-O :

Additionally, I would support adding a regulation which explicitly mentions the option to disqualify/ban competitors when they intentionally ignore 1e1 and 1f1.

You can already disqualify a competitor with: 2k5) Fails to abide by WCA Regulations during the competition.

If we need an explicit case for every way a competitor can intentionally ignore some regulation, 2k will become very long :-)

[Laura-O]

You can already disqualify a competitor with: 2k5) Fails to abide by WCA Regulations during the competition. If we need an explicit case for every way a competitor can intentionally ignore some regulation, 2k will become very long :-)

I am not searching for ways to disqualify someone from a competition, my suggestion was to clarify consequences in the specific regulation as competitors obviously do not take this job as serious as we want them to...

There are other regulations which explicitly mentioning penalties (e.g. 2g3)), and I think it makes sense to do this with scrambling/judging as well. This does not make 2k longer, it is basically the addition of one sentence to these regulations.

[Goosly]

Ah, I see, so your suggestion was to add this clarification to 1e1 and 1f1. I misunderstood!

[HaysCubing]

So I'm a bit late and I'm not an organizer so I'm not too helpful in terms of the logistics of how this would be implemented, but I feel like this is an important topic to share my thoughts on.

I generally agree with Lucas' ideas and where his conversation with Laura is going, but wanted to reinforce the argument that scramble checking top solvers is reasonable and necessary. HippolyteM seemed pretty adamant that adding scramble checks for top solvers is ridiculous and sad to see in the WCA, and I think this is definitely incorrect as it ignores what is actually going on in cubing right now.

I posted this in "Cyoubx's Friends", a popular cubing Facebook group yesterday:

"I feel like the real issue is that there is a clear double standard within the WCA in regards to misscrambles.

If you get a record/significant time then the WCA will check immediately to make sure it was a correct scramble and penalize you if it was not. I'm generally fine with this since we want to preserve competitive integrity.

The unfairness comes in the fact that despite obviously treating high profile solves as different/more important (tons of random kids get great times on misscrambles every weekend and they're never checked), the WCA hasn't enacted any measures to make sure top solvers have their cubes scrambled correctly because it's "against the spirit of the WCA," even though they are obviously OK with treating top solves differently in a negative sense.

This is apparent in other aspects of regulation enforcing as well, although it might not be as clear cut, and top solvers (Max and Jeff are more notable examples) are paying for it.

The reality is that it's essentially impossible to not treat top solvers differently. The WCA has become too big for that to be feasible. Even if the WCA tried not to, videos online get scrutinized by fans for any potential rule violation, even if it's trivial and didn't affect the solve.

If top solvers never get misscrambles, then no one will ever get punished for misscrambles since all the other solves are never checked. This is about as "fair" as you can get without ensuring that there are just no misscrambles ever."

[viroulep]

Thanks for your "competitor side" point of view Kevin.

I'd like to point out a few points in there that I feel need to be commented on:

The unfairness comes in the fact that despite obviously treating high profile solves as different/more important (tons of random kids get great times on misscrambles every weekend and they're never checked)

I sincerely believe any Delegate should react on (=investigate) any "unexpected" time for a given person, regardless of their level.

It is true though that high profile competitors tend to get a lot more attention than others.

the WCA hasn't enacted any measures to make sure top solvers have their cubes scrambled correctly because it's "against the spirit of the WCA,"

While I'd agree on the fact that we need to take more measures to make sure any competitor has their cubes scrambled correctly, I believe that - currently - what is holding back Delegates/organizers is to have to dedicate one extra person for scramble checking, and that it would introduce some overhead in the schedule.

We should definitely try to "benchmark" @lgarron proposal on various events at competitions (but actually checking all cubes, not only high profile ones), to get a more precise idea of what kind of overhead we're talking about.

[lgarron]

An update on this: I haven't forgotten about it. In fact, it's a priority for me. @jfly is helping me figure out how to modify all the score card tools to have extra columns at the moment.

[lgarron]

I've started a new proposal at https://github.com/thewca/wca-regulations/issues/640