Lat/Lon swapped in XML files (both cell and WiFi)

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start recording
2. Export session, keeping files
3. Examine XML files produced

What is the expected output? What do you see instead?
In the gpx element of both the cell and WiFi files, the lat attribute reports 
longitude and lon reports latitude. The internal DB has correct values, 
corruption seems to happen on export.

What version of the product are you using? On what operating system?
Different builds of Radiobeacon, on Cyanogenmod 10.1.2 and 10.1.3-RC1

Please provide any additional information below.
A lot of bogus data has probably been collected this way (I must have added 
plenty of data just off the Somali coast :-). That data can be fixed to be 
usable (so we didn't collect it for nothing.)

An updated version of Radiobeacon with the issue fixed must report a different 
version number in its XML output (e.g. swver="00.6.01" instead of 
swver="00.6.00") to allow for identification of files affected by the bug. This 
cannot be done by just looking at the data (there are some ways to do this but 
they won't catch everything.)

Also, a bunch of bogus data has probably been uploaded to the server that way. 
Therefore data on the server must be analyzed and anything collected with 
Radiobeacon needs to get its lat and lon swapped.

Since we never know for how long someone will use a buggy version of 
Radiobeacon, that code would need to stay on the server permanently and fix any 
Radiobeacon 00.6.00 file that comes in.

We may be lucky as imports on the server seem to have been unavailable for a 
few weeks (since about August 1), so maybe we don't have any bogus data up 
yet...

Original issue reported on code.google.com by mich...@vonglasow.com on 1 Sep 2013 at 10:16

[GoogleCodeExporter]

PS: GPX files have correct lat/lon.

Original comment by mich...@vonglasow.com on 1 Sep 2013 at 10:47

[GoogleCodeExporter]

PPS: a fix for the database loader script is at
http://www.gitorious.org/openbmap/openbmap/commit/3cf6512823c83223d88ada15a6d6c5
deae67f624

It won't correct data already in the DB but will swap lat/lon for incoming 
files if swid and swver match.

Original comment by mich...@vonglasow.com on 1 Sep 2013 at 11:34

[GoogleCodeExporter]

for the sake of completeness: 00.6.01 (XML with sanitization fix) also suffers 
from the lat/lon bug. Fixed database loader script at
http://www.gitorious.org/openbmap/openbmap/commit/ca3d1e59ed616d1311f5f687d40d98
01e92208cd

Original comment by mich...@vonglasow.com on 1 Sep 2013 at 1:02

[GoogleCodeExporter]

I just see that picking the swver is a bit more complicated due to the way it 
is generated: The swver is stored in the db at the time of the recording and 
matches the version of the app with which the data was *collected*. However, 
lat/lon are mixed on *export*.

If we push out a new version of the software which fixes this bug and 
introduces a new swver, this will work fine for data collected with the new 
version. However, data collected with an old version but uploads it with the 
new version will report the old version number but not suffer from the lat/lon 
bug - and we need to clearly identify files which suffer from that bug.

My suggestion: introduce pseudo version numbers. When exporting data collected 
with an old version, "bump" its version number to one of the pseudo versions, 
else report what is in the DB. The client version would, obviously have to skip 
some numbers. Since we are now at 00.6.02, the new client would get 00.6.06 and 
introduce the following version numbers:

00.0.03 - collected with 00.6.00, exported with 00.6.06 or later
00.0.04 - collected with 00.6.01, exported with 00.6.06 or later
00.0.05 - collected with 00.6.02, exported with 00.6.06 or later
00.0.06 - collected and exported with 00.6.06

Original comment by mich...@vonglasow.com on 2 Sep 2013 at 10:36

[GoogleCodeExporter]

A new version that fixes the lat/lon bug and bumps version numbers for old data 
as described above is available at:

http://www.gitorious.org/openbmap/openbmap/commit/cb77e36dbfb10e33b03334824f93c8
ad4404f511

DB loader scripts were updated once more to reflect the fact that 00.6.02 is 
also buggy. Commit is:

http://www.gitorious.org/openbmap/openbmap/commit/4d5b97d72ab0596f5d6e460ea048c2
0871f6e645

Original comment by mich...@vonglasow.com on 2 Sep 2013 at 12:55

[GoogleCodeExporter]

Correction to the above: of course, 00.0.0x should be 00.6.0x. Code in the 
above commits is correct :-)

Original comment by mich...@vonglasow.com on 2 Sep 2013 at 1:04

[GoogleCodeExporter]

As a immediate action I have removed all cells files from the processing queue 
to fix the at a later stage. I will ask Mick to do the same for wifi files, so 
we should be safe that no wrong data will make it to the db.

Also I'll ask Mick to block version 00.6.0 from further upload.

Will have a look at the scripts later..

Original comment by wish7code on 2 Sep 2013 at 1:28

• Changed state: Started
• Added labels: Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Great, thanks. In the meantime I have opened a ticket for the server part at 
https://sourceforge.net/p/myposition/bugs/53/

Let me know if you need any help with data cleanup.

Original comment by mich...@vonglasow.com on 2 Sep 2013 at 3:43

[GoogleCodeExporter]

Should be fixed with latest release.

Original comment by wish7code on 3 Sep 2013 at 7:25

• Changed state: Fixed

[GoogleCodeExporter]

Just saw it... looking at the code, though, I wonder what will happen to data 
that I have collected with the old version but not yet uploaded.

As I understand it, it would get uploaded with lat/lon fixed but still 
reporting one of the buggy versions (the one used for collecting).

Which means I can't export the data I collected over the past two days (which 
happens to be a lot), as the server will reject it. Not nice...

Original comment by mich...@vonglasow.com on 3 Sep 2013 at 7:43

[GoogleCodeExporter]

Based on your work I added a small script to repo, which directly corrects the 
upload files on the server. Thus we don't have to correct the affected files on 
each download.

r8a02c65a08d1 added a python script which heals the lat/lon bug in 00.06.00, 
0.06.01 and 0.06.02 in the xml upload files. The healed xml files will have 
version number 00.06.10. 

As the cleanup is performed server-side, there is no need to apply this script 
locally.

Original comment by wish7code on 5 Sep 2013 at 9:44