Some Android 2.3.x devices can't sync

[hssm]

Originally reported on Google Code with ID 372

What steps will reproduce the problem?
1. Open my deck in Anki or Ankidroid
2. Sync it (the first time its working)
3. do some reviews
4. do a Sync in Ankidroid -> Error

What is the expected output? What do you see instead?
A message that sync was not succesfull.

What version of AnkiDroid are you using? On what version of Android?
0.5 beta 20

Please provide any additional information below.

Reported by TAFKAAZ on 2011-02-01 15:30:58

---

• Attachment: Sauseschritt1.anki

[hssm]

Thanks for uploading the deck! I'm looking at this issue.

Reported by inigo.aldana on 2011-02-01 19:27:01

• Status changed: Accepted

[hssm]

Reported by inigo.aldana on 2011-02-01 19:27:22

• Labels added: Sync

[hssm]

I managed to replicate this problem once, but not again, although I'm doing exactly
the same thing.

I suspect it might be related to the ankiweb.net upgrade that apparently just took
place, or poor network connectivity on my part.

But I did discover that our syncing procedure misinforms about the success or failure
of the full uploads and downloads. Although my full upload failed, it still reported
it as successful.

I will add more appropriate messages shortly.

Reported by inigo.aldana on 2011-02-01 19:54:09

[hssm]

Thank you. 
I tried it again but still no luck. It allways fails. ("Fehler: Deck wurde nicht synchronisiert")
Also I removed my deck from ankiweb and reupload it again. The upload first works but
sync again fails.

Its confusing because I have three other decks and sync is working for them.

Reported by TAFKAAZ on 2011-02-01 20:12:45

[hssm]

Hmm, maybe it's something specific to your Android version or your locale.

Is it possible to get the log using this app called LogCollector and attach it here?

Thanks
Kostas

Reported by inigo.aldana on 2011-02-01 22:34:22

• Status changed: WaitingForFeedback

[hssm]

Some instructions on how to do that if you're willing to:

- Download and install the app Log Collector from the market.
- Do the procedure in AnkiDroid that leads you to the error message.
- Close AnkiDroid and run LogCollector. It will try to collect the logs. Save or copy
paste the result to a text file and attach it to this thread.

In the meantime, I'll try to add a few more error checks in the syncing procedure.

Thanks,
Kostas

Reported by inigo.aldana on 2011-02-01 22:37:37

[hssm]

yes, here is the log.

Reported by TAFKAAZ on 2011-02-01 23:26:53

---

• _Attachment: log_android.txt_

[hssm]

Thanks again.

One more thing, what version of android does your device have and how much memory does
it have?

I see the error happens while decompressing the reply from the server. So I think it's
either the case that the server sends something unexpected, or that the memory fills
up while decompressing in memory. 

Reported by inigo.aldana on 2011-02-02 00:33:16

[hssm]

My device is Nexus One with android 2.3.1 (Gingerbread Mod) and 512MB Memory.
I'm not running many apps so I guess there is enough Memory avaible.

Reported by TAFKAAZ on 2011-02-02 16:55:26

[hssm]

Hmm, I have the same device, but I still can't replicate.
The only difference is that I'm running 2.2

I suspect this might be the problem with the malformed payload to the server that Damien
was complaining about. In the next couple of days I plan to add some much needed error
checks in syncing and a bit more informative logs, so we'll be able to figure this
one.

Your locale is german right? Just in case it's something to do with locale formatting,
do you mind switching to english for a bit and trying syncing while anki is with english
language?

Cheers,
Kostas

Reported by inigo.aldana on 2011-02-02 17:08:21

[hssm]

If you post your ankiweb username I'll check the logs.

Reported by second@ichi2.net on 2011-02-02 18:31:12

[hssm]

I changed the locale to english but it I didn't solve the error.
Today, I also tried with beta24.
My ankiweb username is "moonbeaz"

Reported by TAFKAAZ on 2011-02-03 16:53:36

[hssm]

Thanks!

I went in detail through the log and I see there is something weird. The history field
in JSON is stopping without closing the brackets.

I'm not sure if it's artifact of the LogCollector, or our custom JSON print routine,
or indication of a drowned exception.

Reported by inigo.aldana on 2011-02-03 17:03:45

[hssm]

I just found out we had in place this:

After trying to sync and failing, there should be a file in your SD card:
/sdcard/payloadAnkiDroid.txt

Would you mind attaching it to this thread too?

This a string representation of what was sent to the server. This way we can see for
sure if there is any bad format involved.

Reported by inigo.aldana on 2011-02-03 17:09:25

[hssm]

Sorry the filename is /sdcard/payloadAndroid.txt

Reported by inigo.aldana on 2011-02-03 17:11:36

[hssm]

yes, I think this is the file you need

Reported by TAFKAAZ on 2011-02-03 17:14:30

---

• Attachment: payloadAndroid.txt

[hssm]

Excellent :)

Reported by inigo.aldana on 2011-02-03 17:16:01

[hssm]

That's right after a failed sync right?

Kostas

Reported by inigo.aldana on 2011-02-03 17:18:01

[hssm]

I believe yes. 
Well, I removed the old file and sync again.
this file is right after a failed sync.

Reported by TAFKAAZ on 2011-02-03 17:34:20

---

• Attachment: payloadAndroid.txt

[hssm]

WebApp Error: <type 'exceptions.ValueError'>: Expecting , delimiter: line 1 column 28808
(char 28808) 

Reported by second@ichi2.net on 2011-02-03 18:48:19

[hssm]

Thanks Damien,

Just a quick question if you know. The character 28808, does it refer to the 28808th
byte or the 28808th character? Since we have multibyte characters in the payload, that
could be different.

Reported by inigo.aldana on 2011-02-03 20:40:49

[hssm]

Nevermind,

I've reconstructed the sent JSON object from the payloadAndroid.txt file and went to
the 28808th character and I can see the format error:
Inside the list of reviewHistory entries:
...[-3655243747810136514,1.296559392537E9,0,26.12942077498687,3,-0.0142),2.4882600748003947,2.4882600748003947,9,50.7960000038147,6,3]...
That parenthesis there after -0.0142 is what is causing it.

Only I have no clue as to why it ends up there...
The deck clearly has a float in that column: -0.0142862501188561
and we retrieve it and print it as a double (like we do with all floats).

Reported by inigo.aldana on 2011-02-03 21:15:09

[hssm]

I can't figure this out. I just can't replicate it and the error doesn't make sense
either.

Could it be because of the Gingerbread mod, like some bug in the JSON classes?

What if you review fewer cards and try to sync? Does it always fail the second try
(which is the first partial syncing)?

If you try this, please send the payloadAndroid.txt file again, I'd like to see if
it's always the same record that gets messed up.

If it only happens for this deck, it's worth trying Fix Integrity from Anki desktop,
although I'm not optimistic.

Reported by inigo.aldana on 2011-02-04 06:15:18

• Status changed: Accepted

[hssm]

Hmmm, the case with issue 347 and the malformed json (probably duplicate of this issue)
is also happening on a device running Android 2.3.1

Any AnkiDroid developer running same version android?

Reported by inigo.aldana on 2011-02-04 06:43:28

[hssm]

if the data on disk looks correct I don't think a DB check will help I'm afraid

Reported by second@ichi2.net on 2011-02-04 06:59:00

[hssm]

Issue 347 has been merged into this issue.

Reported by second@ichi2.net on 2011-02-04 07:32:10

[hssm]

If this is a bug in Android 2.3.1 would it be interesting what rom is used in issue
372/347? Just to check if this is a bug in Android-source or in the rom?
I'm using Oxygen2 RC7 (http://forum.xda-developers.com/showthread.php?t=829734) on
a HTC Desire.

I also downloaded and installed ankidroid0.5beta24 a few hours ago and it was not able
to sync to the server. Is there a possibility to get verbose error output? Ankidroid
just tells me "Syncing failed".

Reported by ImmerNurWollen on 2011-02-04 07:34:29

[hssm]

hmm... I'm using MicroMod AOSP Gingerbread.

Reported by TAFKAAZ on 2011-02-04 12:49:52

[hssm]

@inigo.aldana:
Yes, it always fails when partial sync.
I've done your instructions. See payloadAndroid.txt

Reported by TAFKAAZ on 2011-02-04 13:11:34

---

• Attachment: payloadAndroid.txt

[hssm]

This is issue has become detective work...

Thanks TAFK, the problem appears in the same history entry.

I wonder if I can plug this mod to the emulator.

ImmerNur, do you mind sending the file /sdcard/payloadAndroid.txt that you find in
your SD card right after a failed sync?

Kostas

Reported by inigo.aldana on 2011-02-04 14:59:38

[hssm]

@inigo
payloadAndroid.txt is attached. It was created after reviewing all facts for tody and
a failed sync with ankidroid0.5beta24

Reported by ImmerNurWollen on 2011-02-04 16:00:57

---

• Attachment: payloadAndroid.txt

[hssm]

Same problem:
unexpected closing parenthesis in the delay field of a single reviewHistory entry.

Damien, is there something special about this field? Was it always a float? Or did
it change at some point?

Reported by inigo.aldana on 2011-02-04 16:59:09

[hssm]

I don't believe the reviewHistory table has been changed since it was introduced. But
I thought you said it wasn't a problem with the DB?

Reported by second@ichi2.net on 2011-02-04 17:13:17

[hssm]

Obviously I'm wildly speculating, I just find it suspicious (and encouraging towards
fixing this) that both attempts fail on the same column of the same table. 

Reported by inigo.aldana on 2011-02-04 17:50:37

[hssm]

Something I've tested just today with Anki Version 1.2.6 in Windows 7 64bit and ankidroid0.5beta24.
With a "repaired" deck syncing seems to be working.

Copy the Anki-Deck from the Phone to the Desktop 
Open it in Anki: ERR-0100, automatic sync deactivated
Sync the deck manually to the Anki-Server: works
Review some facts, sync to the server again: Error 
- ankiqt.ui.snyc, line 361 in syncDeck; 
- anki.sync, line212, in applyPayloadReply, 
- Exception: Facts missing after sync, please run CheckDB
-> Check DB, 26 cards without facts found
Sync again, Review some cards, Sync again: works

Download deck with ankidroid, open the deck, review some (5) cards, sync: works

Now sync with Desktop-Anki: works
Review card, sync to Server: works

Sync back to Ankidroid: works
Review cards, sync: works

Sync back to Desktop: works

To me it looks like some problems with the deck on the phone. Please find both deck
attached.
japanisch.anki (original deck from phone)
desire_japanisch.anki (syncable deck from phone)
desktop_japanisch.anki (syncable deck from desktop)

If you need more information feel free to write me an email.

Reported by ImmerNurWollen on 2011-02-06 14:28:17

---

• Attachment: japanisch.anki
• _Attachment: desire_japanisch.anki_
• _Attachment: desktop_japanisch.anki_

[hssm]

I think your issue is unrelated. In recent versions Anki checks to make sure the DB
is not corrupt (cards with missing facts) after every sync, to ensure errors don't
propagate. If the problem crops up again then it's a cause for concern, but I suspect
it was simply a bug with a previous version or a bad sync which caused the problems
with your deck, and it is unlikely to repeat.

Reported by second@ichi2.net on 2011-02-06 14:54:34

[hssm]

Would be nice. This still leaves the problem with the old deck on the phone and the
problem Inigo mentioned in Comment 32.
Anyway if a workaround is to create a new deck from the old file to get sync working
again it's fine with me. I will test the syncable deckt during the week.

Reported by ImmerNurWollen on 2011-02-06 15:00:17

[hssm]

after upgrading to android version 2.3.2 the problem don't appear anymore.

So I can confirm it's defect in 2.3.1.

Reported by TAFKAAZ on 2011-02-19 18:23:39

[hssm]

Excellent news, thanks for letting us know :)

Reported by inigo.aldana on 2011-02-19 18:35:23

[hssm]

Problem also experienced by Marcel on the mailing list:
https://groups.google.com/group/anki-android/msg/c23a5518195458a0
He uses cyanogenmod 7 RC1 (I haven't checked whether it is really 2.3.1-based)

Damien suggests the possibility of disabling sync for 2.3.1 devices.

Reported by nicolas.raoul on 2011-02-24 07:05:51

[hssm]

I can confirm that. Syncing on Cyanogenmod 7 does not work

Reported by NorbertNagold on 2011-02-24 08:44:44

[hssm]

Is there a way to run this mod on emulator?
I don't want to root and put this mod, just to debug this issue.

If a developer has this mod, I'd really be interested to know whether the problem appears
in the same spot:
In reviewHistory entry in the json object found in file /sdcard/payloadAndroid.txt,
one of the entries has on the 6th value an unexpected closing bracket, see example:
...[-3655243747810136514,1.296559392537E9,0,26.12942077498687,3,-0.0142),2.4882600748003947,2.4882600748003947,9,50.7960000038147,6,3]...

If the error is consistent, we'll just make a workaround, how hard can it be?

Cheers,
Kostas

Reported by inigo.aldana on 2011-02-24 14:30:48

[hssm]

They just released RC2 for Milestone today. I installed it and guess what...the problem
is gone! I guess it's based on 2.3.2 too (but not sure about that).
I didn't test it very much, just synched 2 times with 2 or 3 revisions inbetween on
each side. I'll test it again with more revisions and full synch tomorrow and tell
you if there are still any problems.

I guess there may be still some cyanogenmod versions out there who have the bug (as
if I understand it right every device has its own cyanogenmod, even if they kinda rely
on the same code), but probably they'll all commit to the new code sooner or later
and that won't be a problem anymore.

Cheers,
Marcel

Reported by Murmel.vienna on 2011-02-24 16:59:57

[hssm]

I hope to get some time to get to the bottom of this, but here's something useful for
devs: http://wiki.cyanogenmod.com/index.php?title=Compile_CyanogenMod_for_Emulator

Reported by inigo.aldana on 2011-02-24 21:11:10

[hssm]

Personally I'd be worried that the issue is isolated to a single occurrence, and can
be fixed without accidentally breaking things - but if you think you can do it, a fix
is better than a warning I guess.

Reported by second@ichi2.net on 2011-02-24 23:29:19

[hssm]

Reported by second@ichi2.net on 2011-02-25 05:46:08

[hssm]

I am running stock 2.3.2 Gingerbread on Nexus S.
Ankidroid 0.5.1 and Anki Desktop 1.2.7.

I also experience the problem, that I can make a full sync from Ankidroid, but after
some reviews, the sync returns "Error: Deckhas not been sinchronized".

If I manually delete the existing deck in Ankiweb, the enxt full sync works (as usual),
and somthimes also the first couple of syncs after short reviews work.
Then, all of a sudden, the problem is back.      :(

Reported by andreas.rossband on 2011-02-25 08:25:39

[hssm]

Update: I was wrong, I'm still affected by that bug. I thought I wasn't because yesterday
I only did really small synchs (2 or 3 cards). The problem is not how often you synch,
I think it's actually only affecting full synch and I also think only full synch when
the ankimobile side has changed is affected (full synch when the Ankidesktop side has
changed is working fine). If you want to make a workaround, it's that code you need
to take a look at.

I'm gonna try to get you that payload file so you have more test cases!

Cheers,
Marcel

Reported by Murmel.vienna on 2011-02-25 14:02:27

[hssm]

here it is (finally)

Reported by Murmel.vienna on 2011-02-25 14:34:39

---

• Attachment: payloadAndroid.txt

[hssm]

Thanks Murcel!

Reported by inigo.aldana on 2011-02-25 14:38:08