Map layers disappear after some time of logging

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start logging.
2. Go to map view.
3. Drive around town for some time (~30-60 minutes).

What is the expected output? What do you see instead?
Expected output: Map scrolls and displays layers (map, WiFi catalog, track, 
WiFis from current session).
This works for some time (about 30-60 minutes or 1500-3000 WiFis). At a certain 
point, refreshes will disappear gradually: First no new WiFis will be shown 
(though the counter on the first tab still goes up, hence new WiFis are still 
being found). Then the track will not be updated - the section that has already 
been drawn stays, but the arrow "detaches" itself and new movements are no 
longer redrawn. Eventually the WiFi catalog and map layers stop updating and 
the screen goes black as the already-updated area scrolls out of view. 
Scrolling works to the end, it is just that data updates are not being handled 
any more.

What version of the product are you using? On what operating system?
Afaik rcd0f69c46f5b, Cyanogenmod 10.1.3-RC2.

Please provide any additional information below.
Stopping and resuming the log will rectify this for some time, but symptoms 
will reappear after an even shorter period of time. Yesterday I stopped logging 
and started a new log, after which things went back to normal. Looks like some 
resource isn't released properly and is exhausted eventually.

Original issue reported on code.google.com by mich...@vonglasow.com on 18 Sep 2013 at 10:11

[GoogleCodeExporter]

Just tried r7f23a683d2b3 and did not notice any more instances of the map layer 
disapearing (sessions were shorter than before, though.)

However, WiFi catalog update is becoming an ever greater issue. Often the 
catalog layer will stop updating completely. Regions that have been rendered 
will usually continue to show on the map but nothing new will be drawn as new 
regions scroll into view. Sometimes even parts of already rendered areas will 
blank out.

This may have to do with the amount of data in the catalog (I've imported the 
WiFis I have logged over the past few weeks, so data in my area is quite dense 
as I've been mapping entire neighborhoods over the past weeks), but it will 
become an issue as the amount of data in the WiFi catalog increases. Densely 
mapped cities (Paris) may already experience issues.

Original comment by mich...@vonglasow.com on 24 Sep 2013 at 10:16

[GoogleCodeExporter]

That's strange: I see the wifi catalog objects even earlier than the session 
objects.
I also checked indices twice, but they seem to be good.

You mention you've imported the wifis you have logged over the past few weeks: 
did you consider some kind of clustering?

The original wifi catalog uses the server-side clustering, so there's exactly 
one point for each wifi. If you're importing raw data, the number of points 
might become really huge.. 

Original comment by wish7code on 25 Sep 2013 at 9:15

[GoogleCodeExporter]

I do use some clustering as well. I average over all lat and lon values for a 
given BSSID (so each BSSID shows up only once) and also ignore BSSIDs already 
in the catalog. That gives me exactly one point per WiFi, but that is still 
quite a lot.

One thing I have noticed today is that often the WiFI overlay will show just in 
the southern half of the map, and periodically a stripe on its northern 
boundary will be removed. (See screenshot.)

Could serialization be an issue here, similar to the current session?

Another idea I had: round lat and lon to 4 decimals (accuracy of ~10 meters, 
which should be sufficient for visualization) and return just one record per 
position. For the map view, it doesn't really matter how many WiFis are in a 
given location, the important thing is that it's been surveyed and I should 
better concentrate my mapping efforts on some other area...

Original comment by mich...@vonglasow.com on 25 Sep 2013 at 9:53

Attachments:

• Screenshot_2013-09-25-22-55-33.png

[GoogleCodeExporter]

> Could serialization be an issue here, similar to the current session?
No, wifi catalog objects aren't de-serialized, it's just raw access to the wifi 
catalog database.

I've overlooked something so far: some time ago, I implemented a hard-limit of 
2500 objects to be displayed.

(Reference:
http://code.google.com/p/openbmap/source/browse/android/src/org/openbmap/utils/W
ifiCatalogMapObjectsLoader.java#138)

Do you have a build environment ready to play around with that limit (removing 
it completely, or increasing the limit)? I would be interested, whether the 
problem is gone then and how Radiobeacon performs when there's no such limit in 
place.

In case you don't have a build environment in place, just drop me a line, so I 
would provide binaries..

Original comment by wish7code on 26 Sep 2013 at 11:52

• Changed state: Started

[GoogleCodeExporter]

Or three lines above, as you suggested: 

replace sql by

SELECT round(latitude,4) as lat, round(longitude,4) as lon FROM wifi_zone WHERE 
(latitude > ? AND latitude < ? AND longitude > ? AND longitude < ?) GROUP BY 
lat, lon

Original comment by wish7code on 26 Sep 2013 at 12:27

[GoogleCodeExporter]

Nevermind.. Have a look at r8879fc8c6fe0 which implements both..

Original comment by wish7code on 26 Sep 2013 at 2:20

[GoogleCodeExporter]

Thanks a lot, I'll give it a shot.

Today, when going north, I noticed that the WiFi layer was OK for approximately 
the lower half of the screen, whereas the upper half was bare of WiFis. So it's 
probably related to some limit on the objects returned, either a bounding box 
or the number of objects. I'll try the new build.

Original comment by mich...@vonglasow.com on 26 Sep 2013 at 3:34

[GoogleCodeExporter]

Tested it at the same zoom level shown in the screenshot, and I got a complete 
WiFI overlay again. So I can say that at the zoom level/resolution I used 
(displaying an area of about 800×800 m), the combination of rounding, 
clustering and allowing up to 5000 objects will work for a completely mapped 
inner city area.

Original comment by mich...@vonglasow.com on 26 Sep 2013 at 6:47

[GoogleCodeExporter]

Original comment by wish7code on 28 Sep 2013 at 6:58

• Changed state: Fixed