gnosygnu
commented
7 years ago Hi! Thanks for the detailed ticket info.
a) If XOWA and a few other apps get minimized, the download/unzip process halts (even if Marshmallow battery optimization is turned off for the app). The app may need to run in the foreground via having a notification or being a service to avoid this...I'm not sure on the details here though.
That stinks. There's actually code to keep XOWA open via the WAKE_LOCK. See https://developer.android.com/training/scheduling/wakelock.html
The WAKE_LOCK works on my two test devices, though I've only done a handful of 10+ hour downloads. That said, I'm not surprised to hear that it's not working on others given what's involved.
A few questions:
- What device are you using? I have an LG G4 (Marshmallow) and a BLU Studio 4C (Lollipop)
- Where are you geographically / how is your network connection? I'm in the NYC metro area and I have a pretty good network connection (so I don't get many drops)
b) Bad checksum or connection loss errors do not trigger retries, so after waiting X hours one may find out that the download stopped 20 minutes in. The PC app seems to retry a bit harder (though it still stops in too many cases IMO).
Yeah, I thought checksum errors would be so rare that it wasn't worth retrying. I'll add that to a future release
I'm surprised that connection losses don't trigger retries. I put some code in there last month to retry 6 times over a minute for a connection loss. Keep in mind the exact same code runs on both Android and PC, so I'm not sure why you're seeing different results. It's possible that the Android session was victim of a longer timeout than the PC one.
c) It might just be my aging pre-UHS-1 sdcard interface, but unzip is as slow as 20mbps downloads. It might be useful to download the next segment while the prior one unzips, but this would mostly be "icing on the cake" to make things 2x as fast or so. On PC, unzip is far faster of course, so this would have less benefit.
I'll look into that, but it may not be meaningfully faster since it's still two concurrent operations (download and unzip) on one CPU and one SD card. At any rate, I should take a look and confirm one way or the other.
Furthermore, unlike on PC, all these errors and manual retries sometimes result it truncated/bad files (in spite of the "verify" step). For example, 95%+ of enwiki images were broken until I copied my PC's "core" image xowa file (as opposed to the many "nsxxx.dby" ones) over the tiny Android "core" image xowa file (the later was clearly broken, being only 70Kb or so). That made the vast majority of images start working. I guess that is a "master"/index file for images. I suspect that as I copy over more xowa files (over USB) with ones downloaded on my PC, the remaining broken thumbnails will be fixed too.
Honestly, that's really unexpected. There's two separate checksum operations (one for the download and one for the unzip) so I'm surprised a file managed to pass both and still be truncated / bad.
Your Google Play review mentioned that you were stuck at 3 of 66 . This should mean that the first two files (1 - 2) were downloaded and verified, but the rest (3 - 66) were not. However, XOWA does create files if it tries to open an image in a missing file and it doesn't exist. For example, say XOWA needs an image from database 65 but database 65 doesn't exist. It still creates a small 70 kb file for database 65 anyway.
This is a small bug with the SQLite library, that I should be able to avoid by calling a different API. If that's the case, I'll fix that for a future release.
Finally, on a separate point, the Google Play review did mention the wide navboxes at the bottom. I'm looking to add a config change for it in a release within the next week. I'll post a comment here when I do release it.
Let me know if there's anything else I missed above. Thanks again for all the detail.
AaronSchulz
markusmaresch
App xowa downloads are theoretically convenient, but they end up requiring a lot of attention as: a) If XOWA and a few other apps get minimized, the download/unzip process halts (even if Marshmallow battery optimization is turned off for the app). The app may need to run in the foreground via having a notification or being a service to avoid this...I'm not sure on the details here though. b) Bad checksum or connection loss errors do not trigger retries, so after waiting X hours one may find out that the download stopped 20 minutes in. The PC app seems to retry a bit harder (though it still stops in too many cases IMO). c) It might just be my aging pre-UHS-1 sdcard interface, but unzip is as slow as 20mbps downloads. It might be useful to download the next segment while the prior one unzips, but this would mostly be "icing on the cake" to make things 2x as fast or so. On PC, unzip is far faster of course, so this would have less benefit.
Furthermore, unlike on PC, all these errors and manual retries sometimes result it truncated/bad files (in spite of the "verify" step). For example, 95%+ of enwiki images were broken until I copied my PC's "core" image xowa file (as opposed to the many "nsxxx.dby" ones) over the tiny Android "core" image xowa file (the later was clearly broken, being only 70Kb or so). That made the vast majority of images start working. I guess that is a "master"/index file for images. I suspect that as I copy over more xowa files (over USB) with ones downloaded on my PC, the remaining broken thumbnails will be fixed too.