1.8.9.1 DISCUSS field testing sites - bad dates showing up in 2018, too

[eyarzebinski]

Prior to this week's data download, the battery-dies-and-resets-date-to-2000 issue explained all of the weird dates showing up in the data.

Now I'm seeing dates from Jan 2018 in the data (1/12, 1/13, 1/14, 1/15, 1/17). These show up in two villages, 128 and 131. In village 128, 1 tablet is affected by this issue, with 14K rows out of 550K total village rows (~2%) with the bad 2018 dates. In village 131, 1 tablet is affected, with 25K out of 150K total village rows (~16%) with the bad 2018 dates.

This doesn't affect ordering since Judith and I were able to order the sessions independent of the dates. However, the bad dates encroaching on 2018 makes me nervous about the other 2018 dates. There are a few (and only a few so far) analyses that do take dates into account, such as "days since deployment".

@JackMostow and @kevindeland, given that it's 2 total tablets, do you see a benefit of trying to fix this? Not sure how much time it would take.

[JackMostow]

@eyarzebinski - Does the wrong-date issue affect our beta sites? If not, fixing it in code drop 2 won't affect our chances of winning, just the value of the data for other purposes.

[eyarzebinski]

I have not seen this in the beta sites so far.

[JackMostow]

I'll ask XPRIZE if anyone else has encountered this issue or knows the reason.

[JackMostow]

@eyarzebinski - I just posted the following query to the new tech lead. Can you confirm that both the tablets whose logs (sometimes) have January 2018 timestamps are using version 1.8.9.1? That would rule out the possibility that two tablets with the previous version somehow didn't get replaced with new ones.

Jo - Welcome! As you may have heard, the Pixel C tablets reset their clocks to the year 2000 when they run out of power and recharge other than on a server with NTP. Fortunately, those bogus dates are easy to spot. However, our data analyst reports a mystery I'm hoping you may be able to shed light on:

"In the RoboTutor logs, she is "seeing dates from Jan 2018 in the data (1/12, 1/13, 1/14, 1/15, 1/17). These show up in two villages, 128 and 131. In village 128, 1 tablet is affected by this issue, with 14K rows out of 550K total village rows (~2%) with the bad 2018 dates. In village 131, 1 tablet is affected, with 25K out of 150K total village rows (~16%) with the bad 2018 dates."

This version wasn't deployed until after April 2018, so it shouldn't have timestamps before then.

1. Are any other teams encountering this issue?

2. Might it be caused by mis-set clocks on servers in villages 128 and 131? It seems unlikely that they'd affect only one tablet in each village, unless they're on servers used only rarely. Do those (or any) villages have more than one ftp server?

3. Any other hypotheses?

Thanks! - Jack

[JackMostow]

@eyarzebinski - BTW, I've been meaning to ask: How did you manage to reconstruct the order of logs despite the bogus timestamps?

[JackMostow]

@kevindeland - Jo asked "As a 3rd hypothesis: since there is only one tablet in each of those two villages affected, could it be that the child somehow edited the tablet's date settings?"

Do you know whether the kiosk mode you used prevents such editing?

[eyarzebinski]

@JackMostow I've confirmed that the tablets showing bad 2018 dates have version sw.1.8.9.1.

For resolving bogus timestamps, I realized that since we have a set curriculum, we can compare the sequence they progress through and use that to order the sessions in the right tutor sequence. Judith wrote me a python script to calculate it, and so that outputs the order of each session.

[JackMostow]

Clever, and may be the best possible, but:

1. What if they get demoted to a lower level?

2. What if two sessions are limited to different content areas so they could be in either order? Or do we figure that their order doesn't matter in that case?

[JackMostow]

@nirmalpatel - Please post:

• for @eyarzebinski: the rows logged as 6th and 7th items of a 5-item sequence
• for @judithodili: the list of data sources with over 10 items that should be marked "rand" to limit to 10. Thanks. - Jack

[JackMostow]

@nirmalpatel - Oops, I just saw your 7/31/2018 6:10pm email. - Jack

[judithodili]

Doesn’t bubble pop have the 3 strike rule per question? --

Regards, Judith Odili Uchidiuno www.judithu.com

[nirmalpatel]

@judithodili I guess that would increase the attempt number for the items but not the problem number? In the data, we are seeing that the CF (Problem Number) field is going > 10. And one of the rand levels has 31 max problems, which is more than 10 * 3 (if 3 strike rule gives 3 chances per question...)

[JackMostow]

@judithodili - #342 describes the cause and cure of this bug, but I didn't know how frequent it was. @nirmalpatel's table documents cases where it occurs, although not necessarily every single one -- just those encountered by at least one kid who attempted more than 10 items.

[eyarzebinski]

@JackMostow asked: Question about the bogus January 2018 dates — do they start on the same dates and cover the same approximate date ranges?

[eyarzebinski]

They start on different dates. Village 128 has bad 2018 dates from 1/11 to 1/14, and Village 131 has bad 2018 dates from 1/14 to 1/18.

[JackMostow]

Curiouser and curiouser! Do they have any dates from 2000? If so, with what range? If someone changed the year from 2000 to 2018 but not the month or day, I'd expect some dates from 1/1/2000 - 1/11/2000 on the tablet from site 128 and from 1/1/2000 - 1/14/2000 on the tablet from site 131.

[eyarzebinski]

The village 128 tablet (6116002524) has data logged on 1/2/2000 - 1/6/2000 and then again 1/11/2018 - 1/14/2018. No data is logged for any other dates.

The village 131 tablet (5A27001580) has data logged only on 1/14/2018 - 1/18/2018. No data is logged for any other dates.

[JackMostow]

I just posted this weirdness to Jo via Slack, and added:

I wonder if these tablets have something wrong with them. Perhaps your field staff can check? Thanks. - Jack

[JackMostow]

@eyarzebinski - I had mistakenly thought that 1.8.9.1 calls NTP to set the clock. It doesn't. The clock is set manually by XPRIZE staff when they configure the tablets. Apparently they mis-set two of the tablets, which the field staff were going to replace by now. If you check their tablet ids in the latest data, you may see evidence of starting over with the placement test.

Since this information solves the mystery, I'll close the issue. But I added 1.8.9.1 number log files sequentially #366 and 1.8.9.1 use NTP to update tablet clock? #367.