Download fail: Size mismatch

[obifur]

I tried it several times but I have not been successful to download the database.

I'm on CM13 nightly, installed Xposed v85 and XposedUnifiedNlp 1.0.4-Experimental. GSM LocationProvider 1.4.5 from F-Droid.

Logfile: info: MCC Filter: 262 info: Keine MNC Filter, nehme alle Netzwerke info: Die URL lautet http://opencellid.org/downloads/?apiKey=dev-usr--1ed5-4725-8e83-e7dfd4b52388&filename=cell_towers.csv.gz info: Datengröße = 538602908 000%: Gelesene Einträge:1000, Gespeichert: 867 000%: Gelesene Einträge:2000, Gespeichert: 1743 000%: Gelesene Einträge:3000, Gespeichert: 2412 000%: Gelesene Einträge:4000, Gespeichert: 3196 000%: Gelesene Einträge:5000, Gespeichert: 4196 000%: Gelesene Einträge:6000, Gespeichert: 5112

...and so on...

-99%: Gelesene Einträge:27869000, Gespeichert: 1859066 -99%: Gelesene Einträge:27870000, Gespeichert: 1859087 -99%: Gelesene Einträge:27871000, Gespeichert: 1859107 -99%: Gelesene Einträge:27872000, Gespeichert: 1859123 -99%: Gelesene Einträge:27873000, Gespeichert: 1859143 -99%: Gelesene Einträge:27874000, Gespeichert: 1859153 fail: Size mismatch fail: Size mismatch 100%: Size mismatch info: Gesamtzeit: 3463000 Millisekunden info: Fertig.

Thx for your effort!!

[n76]

Interesting failure with the "Size mismatch". I am wondering if we are getting mismatch between how big OpenCellId says the file is and how much we can actually read. Can you try it using Mozilla as the data source rather than OpenCellId? (Mozilla and OCID exchange data so you should have pretty much the same database using Mozilla.)

[obifur]

I'm not sure what happened this time:

info: MCC Filter: 262 info: Keine MNC Filter, nehme alle Netzwerke info: Die URL lautet https://d17pt8qph6ncyq.cloudfront.net/export/MLS-full-cell-export-2016-06-18T000000.csv.gz info: Datengröße = 559981916 000%: Gelesene Einträge:1000, Gespeichert: 0 000%: Gelesene Einträge:2000, Gespeichert: 0 000%: Gelesene Einträge:3000, Gespeichert: 0 000%: Gelesene Einträge:4000, Gespeichert: 0 000%: Gelesene Einträge:5000, Gespeichert: 0

...

087%: Gelesene Einträge:23796000, Gespeichert: 1249290 087%: Gelesene Einträge:23797000, Gespeichert: 1249290 087%: Gelesene Einträge:23798000, Gespeichert: 1249290 087%: Gelesene Einträge:23799000, Gespeichert: 1249290 087%: Gelesene Einträge:23800000, Gespeichert: 1249290 087%: Gelesene Einträge:23801000, Gespeichert: 1249290 info: Gelesene Einträge:23801772, Gespeichert: 1249290 info: Gesamtzeit: 3014641 Millisekunden ( 0.127 Millisekunden je Eintrag) 100%: Erstelle Index für schnellen Zugriff. info: Gesamtzeit: 3109617 Millisekunden info: Fertig.

I wonder why he jumped from 87℅ to 100℅.. However there is a lacells.db with 96,8MB now, so probably everything is fine now!? On the screen I see no hint, it just says "last database update today" and "update database"

Edit: Yes, it seems to work. I have a location when I'm offline, even if it's not so accurate!

[n76]

Size estimate is approximate as the web server reports the size of the ZIP file but due to how it is piping the data through a unzip and CSV component the app only knows how many expanded bytes it has read. So we estimate the unzipped size based on about how much we expect the expansion to be.

It seems there is an issue with reading data from OCID, which narrows things down a bit.

Accuracy of your position estimate will depend on phone (mine only reports one tower so that is the location estimate I get) and how well the towers in your area have been mapped.

[n76]

It appears that the ZIP file(s) from OCID are not compatible with what the GZIPInputStream on Android expects:

06-18 11:45:21.256  2541 26269 W System.err: java.io.IOException: Size mismatch
06-18 11:45:21.325  2541 26269 W System.err:    at java.util.zip.GZIPInputStream.verifyCrc(GZIPInputStream.java:268)
06-18 11:45:21.325  2541 26269 W System.err:    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:150)
06-18 11:45:21.325  2541 26269 W System.err:    at java.io.InputStream.read(InputStream.java:162)
06-18 11:45:21.325  2541 26269 W System.err:    at java.io.BufferedInputStream.fillbuf(BufferedInputStream.java:149)
06-18 11:45:21.326  2541 26269 W System.err:    at java.io.BufferedInputStream.read(BufferedInputStream.java:234)
06-18 11:45:21.326  2541 26269 W System.err:    at org.fitchfamily.android.gsmlocation.CsvParser.parseLine(CsvParser.java:80)
06-18 11:45:21.326  2541 26269 W System.err:    at org.fitchfamily.android.gsmlocation.CsvParser.parseLine(CsvParser.java:26)
06-18 11:45:21.326  2541 26269 W System.err:    at org.fitchfamily.android.gsmlocation.async.DownloadSpiceRequest.getData(DownloadSpiceRequest.java:374)
06-18 11:45:21.326  2541 26269 W System.err:    at org.fitchfamily.android.gsmlocation.async.DownloadSpiceRequest.loadDataFromNetwork(DownloadSpiceRequest.java:241)
06-18 11:45:21.326  2541 26269 W System.err:    at org.fitchfamily.android.gsmlocation.async.DownloadSpiceRequest.loadDataFromNetwork(DownloadSpiceRequest.java:53)
06-18 11:45:21.326  2541 26269 W System.err:    at com.octo.android.robospice.request.CachedSpiceRequest.loadDataFromNetwork(CachedSpiceRequest.java:48)
06-18 11:45:21.326  2541 26269 W System.err:    at com.octo.android.robospice.request.DefaultRequestRunner.processRequest(DefaultRequestRunner.java:150)
06-18 11:45:21.326  2541 26269 W System.err:    at com.octo.android.robospice.request.DefaultRequestRunner$1.run(DefaultRequestRunner.java:217)
06-18 11:45:21.326  2541 26269 W System.err:    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:423)
06-18 11:45:21.326  2541 26269 W System.err:    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
06-18 11:45:21.326  2541 26269 W System.err:    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
06-18 11:45:21.326  2541 26269 W System.err:    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
06-18 11:45:21.326  2541 26269 W System.err:    at java.lang.Thread.run(Thread.java:818)

Not sure what to do about this as if it happens early in the process we probably want to abort (like we do now) but if late in the process of building a database it might be nice to keep the records we've already read and entered into the database.

[FoxyBee]

I have the same problem. Tried multiple times and fails every time. Here is my log: lacells_gen_log.txt

Retrying now using only Mozilla as data source.....

Update: Using only Mozilla data source, database was generated successfully. The only item of concern is that the log file indicates the read operation ends at 87% with number of records read = 24064219. The next log entry is "100% creating indicies for faster access"

[zoff99]

+1 i have the same problem. size mismatch, at about 91% it starts counting backwards.

suggestion: can you make an option to just read (and save) until the end of stream, no matter if the sizes match?