Export is unstable

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start monitoring.
2. After capturing a decent amount of data, save.
3. Upload.

What is the expected output? What do you see instead?
Expected: data is exported and uploaded, confirmation of successful upload at 
the end.
Actual:
1. I get a warning message that my client version couldn't be verified because 
either I'm offline, the server is down or my client is too old, and am asked if 
I want to continue at my own risk. (See screenshot.)
2. After continuing, data is exported into a set of ~10 files and upload starts.
3. The logcat (alogcat.2013-08-23-11-26-50+0200.txt) shows errors about some 
files not uploading successfully. Apparently not all the data was uploaded 
successfully.
4. In another instance, I cannot upload the log at all. It exports to file but 
the upload progress bar never moves. Apparently something did get uploaded, as 
I see in the logcat (alogcat.2013-08-23-23-59-19+0200.txt), but I can't tell if 
that's all.

What version of the product are you using? On what operating system?
Built from source, revision c4f48ccb716e, on Cyanogenmod 10.1.2.

Please provide any additional information below.
Is there a particular reason that each log is split up in about 10 files for 
each object type? That makes it a lot harder to keep track of which file got 
uploaded and which one didn't.
Also, at the moment the state of server data is unclear: cell data seems to be 
3 weeks old (judging by the timestamps in the raw data zip file), the zip file 
for raw wifi data is corrupt and can't be opened, and there is no indicator 
showing when the last update took place. (I've opened tickets for that on SF:)

Original issue reported on code.google.com by mich...@vonglasow.com on 23 Aug 2013 at 10:39

Attachments:

• Screenshot_2013-08-23-23-57-35.png
• alogcat.2013-08-23-11-26-50+0200.txt
• alogcat.2013-08-23-23-59-19+0200.txt

[GoogleCodeExporter]

Regarding "1. I get a warning message that my client version couldn't be 
verified because either I'm offline, the server is down or my client is too 
old, and am asked if I want to continue at my own risk."

I might have an easy quick-fix for this: could you verify, that your wifi is 
not in sleep mode (hit cancel and try again, in the meantime the wifi should be 
ready). I observed this problem, when I haven't used the device for a couple of 
minutes. As a quick-fix we would just wait a couple of seconds, so the wifi has 
time to setup connection

Regarding 3. and 4.: Upload interval might still be to aggressive: files are 
uploaded asynchronously, so several uploads might happen at the same time. So I 
fear server just denies connection in that case. I'm in contact with Mick, who 
runs the server regarding this issue.

Original comment by wish7code on 28 Aug 2013 at 9:43

• Changed state: Accepted

[GoogleCodeExporter]

Regarding 1, I just tried (though I was on mobile data, not WiFi) and got the 
same behavior twice. 

Regarding 3, I played with the database loader scripts the other day and a 
bunch of my exported files woudn't load successfully. One thing I noticed is 
that all the "krank" files I looked at somewhere had picked up a WiFi with an 
ampersand sign in the SSID, which appears as a literal ampersand, not escaped, 
in the XML files. Maybe that has something to do with it.

Regarding 4, I forgot to mention that in this case the app crashed completely. 
I have now taken a look at the logcat (alogcat.2013-08-23-23-59-19+0200.txt), 
the behavior is caused by a android.os.NetworkOnMainThreadException. That 
happens when the main UI thread tries to access the network, which Android 3.0 
and higher don't allow. In Android 2.x this is apparently not an issue. The 
"offending" call is SessionListFragment.isAllowedVersion(). Strangely though, 
it happens only sporadically. The log in question failed to upload on a few 
attempts, but I managed to upload it successfully just now.

Further reading: 
http://www.androiddesignpatterns.com/2012/06/app-force-close-honeycomb-ics.html

Original comment by mich...@vonglasow.com on 29 Aug 2013 at 10:30

[GoogleCodeExporter]

update: the failed uploads in alogcat.2013-08-23-11-26-50+0200.txt were not in 
the "krank" list, the files seem OK.

Original comment by mich...@vonglasow.com on 29 Aug 2013 at 10:43

[GoogleCodeExporter]

Changeset 898b543cd016 addresses this issue with several approaches:

StaleVersionChecker is now using AsyncTask, so at least the 
android.os.NetworkOnMainThreadException is fixed. BTW, this behaviour is really 
strange as my 2.3.3 device indeed never complaint about that, the 4.3 does...

For the wifi uploads, I added a delay of 1 sec after each upload, to give the 
server time to catch up. Note that this will slow down uploads significantly. 
After some testing we could easily adjust smaller delays.

I also opened a new bug for the ampersand thing. Even if it's not causing this 
issue, we have to fix that asap..

Original comment by wish7code on 29 Aug 2013 at 7:26

• Changed state: Started

[GoogleCodeExporter]

Yep, Android 3.0 started getting stricter about that. Doing network operations 
in the UI has always been dicouraged as it can easily block the UI, and 
Honeycomb started throwing exceptions at any app doing that.

About the uploads, would it help to place the files in a queue and have a 
single thread upload one, wait for it to finish, and then upload the next? 
After all, we have only so much bandwidth, so one way or the other, upload time 
is determined by combined size of files to transfer, divided by bandwidth, plus 
some overhead. And doing things simultaneously rather than sequentially tends 
to increase overhead.

If the "one file at a time" approach suffers from delays between the end of one 
download and the start of the next, then how about limiting the number of 
simultaneous connections? Uploads could then be handled by a pool of worker 
threads (as many threads as simultaneous connections).

Original comment by mich...@vonglasow.com on 29 Aug 2013 at 7:40

[GoogleCodeExporter]

Please have a look at Revision 48d9ccf016b2 

It includes the following changes:
- the version checker should now handle wifi-sleep mode much better. If no 
connection isn't available immediately, it waits for a couple of seconds to 
establish wifi connection
- "automatic repair option": on some devices, it helps to switch wifi off and 
re-enable to get wifi connection. User is asked, whether he wants to give it a 
try 
- following your suggestion the number of simultaneous connections is limited 
(see ExportManager around line 176 for details)

As I don't have a mobile data connection (UMTS, LTE) I couldn't test, whether 
the above measures will also help there.

I would like to hear your feedback, whether problems still occur. 

Original comment by wish7code on 3 Sep 2013 at 7:45

• Changed state: Fixed

[GoogleCodeExporter]

OK, I will try.

So far I've tried one upload with 00.7.00, on UMTS (throttled to 64k as I've 
used up my data volume) and got another crash (see issue 8). Will try on WiFi 
tonight.

Original comment by mich...@vonglasow.com on 4 Sep 2013 at 11:38