Data missing from log

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Enable export of GPX file.
2. Start logging.
3. Drive about town for about 40 minutes, collecting some 1800 WiFis and 35 GSM 
cells. (Note that the Nexus S can retrieve only the current cell.)
4. Export.
5. Examine the GPX file and the log files produced.

What is the expected output? What do you see instead?
In a log I created today, trackpoints as well as WIFI waypoints and CELL 
waypoints start appearing at 18:24 and continue until 19:09. When analyzing the 
WiFi XML files, the earliest one starts at 18:57 and the latest ends at 19:09. 
Cell XML files seem to span the entire period (though the serial number in the 
filename is not contiguous, but that is confusing at the worst).

What version of the product are you using? On what operating system?
Revision 8e96a4f883b4, Cyanogenmod 10.1.3-RC1 on Nexus S.

Please provide any additional information below.
Vversions prior to 00.7.00 had a similar bug: they would skip a number of scans 
between XML files (each log covered less than a minute and the next file would 
begin at least 5 minutes after the end of the previous one). 00.7.00 introduced 
the current behavior, with the first 30 of 40 minutes missing and the rest of 
the log data being contiguous.
Also, when re-exporting with 00.7.00 a log that I had created (and previously 
exported) with 00.6.02, I noticed that 00.7.00 produces significantly more 
files for the same log than 00.6.02. 

Original issue reported on code.google.com by mich...@vonglasow.com on 12 Sep 2013 at 7:32

[GoogleCodeExporter]

Michael, unfortunately I was unable to reproduce the problem (just exported 
~4000 wifis) and everything seems fine at a first glance.

Let's see whether we can track down the error:
1) Are all expected wifis persisted in the database (see Wifi Overview list, or 
grab the database directly via adb pull 
data/data/org.openbmap/databases/radiobeacon

2) Does the gpx track show all expected databases (for a quick check, you could 
display the track with http://josm.openstreetmap.de/)

3) Then you wrote '.. the earliest one starts at 18:57 and the latest ends at 
19:09..': I assume, you refer to the gps timestamp within the xml file (e.g. 
<gps time="20130911185600", not the timestamp in the filename (e.g. 
(username_V1_log2013...) Correct?

If you don't mind sharing your logging database, drop me a private mail, so we 
could better analyse this issue..

Last word on 
'though the serial number in the filename is not contiguous, but that is 
confusing at the worst': The server takes the file timestamp as primary key, 
i.e. there can only be one file named username_V1_20130911185700, otherwise the 
existing file is overwritten. This might not be the wisest solution, but we 
can't change it on server-side at the moment. 
So in order to prevent primary key collisions, the file timestamps are 
increased monotonic, thus generating a pseudo-timestamp. We might have to 
reconsider this mechanism. See
http://code.google.com/p/openbmap/source/browse/android/src/org/openbmap/soapcli
ent/WifiExporter.java#553

Original comment by wish7code on 16 Sep 2013 at 10:07

• Changed state: Started

[GoogleCodeExporter]

I just looked at the data again, and it seems like all the data did get 
exported. I must have overlooked the first batch of WiFi files... don't know 
how that happened. I suspected something had gone wrong when the exported data 
didn't show up in the Mapnik rendering I had created.

Serial numbers: I had realized they were timestamps, only now I realized that 
it's just yyyymmddhhmmss with no additional serial number appended (as I had 
assumed before).

My bad... sorry about the confusion. This one can be closed as invalid.

Original comment by mich...@vonglasow.com on 17 Sep 2013 at 10:39

[GoogleCodeExporter]

> don't know how that happened
This (Android) issue might be the cause:

http://code.google.com/p/android/issues/detail?id=38282

There's an annoying Android bug, so that new files won't show up, when 
connected to a desktop pc. Only after a reboot files become visible.

Newer client versions (>= r43738d37092f) address this issue by rescanning the 
sd-card following a successfull export. For details have a look at the 
MediaScanner class.

>  the exported data didn't show up in the Mapnik rendering I had created.
I'm really excited to learn more about that visualization :-) 

Original comment by wish7code on 17 Sep 2013 at 12:01

• Changed state: Invalid

[GoogleCodeExporter]

I'll happily share... where do I send it? I was looking for a wiki on that 
topic which would accept contribution from me, either the Radiobeacon or the 
obenBmap wiki...

Original comment by mich...@vonglasow.com on 18 Sep 2013 at 9:58

[GoogleCodeExporter]

Recently I added a section on possible data uses 
http://code.google.com/p/openbmap/wiki/WorkingWithOpenbmapData

Do you think it would fit there?

If you like, we could also add a separate wiki page (topic visualisation?)

Original comment by wish7code on 19 Sep 2013 at 12:10