User Reported Issue: Correct answer marked as wrong

[phet-steele]

Email follows:

It's difficult to replicate (and admittedly was encountered in a very large noisy class). An example in Level 5. The problem called for an area of 15 with one third light and two thirds dark. The student had a 3 row by 5 square array with the top row light and the next two rows dark. The program called it incorrect twice then showed an array of 5 rows with 3 squares in each with the first column light and the next two columns dark.

The student was quite frustrated.

I hope that helps. We were using Google Chrome, the current version.

[phet-steele]

@jbphet let's consider this a sighting. I managed to get the same prompt and set up the answer the same way with no problem.

[jbphet]

I created a version of the sim that only generates the challenge that the user described for level 5 and played the challenge quite a number of times in the way that the user described and with a number of variations (all the blocks on the top, all on bottom, left, right, various random arrangements). So far, I haven't seen any problems. I've responded to the original email to ask if they could try to duplicate it and send a screen shot.

[jbphet]

The teacher responded, and was able to duplicate the issue, and sent some screen shots. Here they are:

[jbphet]

The challenge shown in the screenshots shown just above is a bit different than the one described in the original problem report, but I can't duplicate the one in the screenshots either. I've created a custom version that only does only this particular challenge configuration on game level 5, and I can create exactly the shape shown in the screen shots and it works just fine. Here's a screenshot from this custom version:

[jbphet]

An observation: In the screenshots sent by the teacher, the background of the "You built" window is white, which indicates that the window code thinks that the answer is correct. Edited screenshot below. This is all pretty mysterious.

[jbphet]

I created a couple of special versions of code that output information to the console that would potentially help to track down this issue and sent links to the teacher who originally reported the issue. She was unable to duplicate the problem with these special versions, so I didn't get any useful data from this.

[jbphet]

I've made some changes to the code that evaluates correctness of the proportional challenges. Previously, this code calculated the proportion by dividing the number of blocks of each color by the total number of blocks, and then comparing the results to the "build spec". Because of floating point variances, there was a threshold for comparison in this calculation. My best theory at this point is that the calculation is somehow a bit different on the machines being used by this teacher and her class, and the comparison tolerance is too low and the answer is evaluated as being incorrect. I've modified the evaluation of these challenges to match the evaluation used in the "You Built" window, which uses a Fraction type and thus compares numerator to numerator and denominator to denominator, so no comparison of potentially irrational numbers is necessary. I plan to work with the teacher to have her test the modified version and see if the problem has gone away.

[phet-steele]

@alynne-phet and @phet-steele couldn't cause the issue after about 15 minutes each.

[jbphet]

I've sent a request to the teacher who originally reported this issue to test the updated version. I'll post the results here if and when I hear back.

[jbphet]

The teacher reported back that she played level 5 of the game several times on v1.1.0 without encountering the error. That's good news, and I'm going to close the issue. I also asked what hardware was being used, and she said, "we were using 10-Zig Virtual machines and Dell Optiplex 380 on Windows 7 Pro. Both were using Google Chrome."

It seems like the issue may well be addressed, closing.