AndNav should avoid using 10000+ 2kb images as data source for maps

[GoogleCodeExporter]

Actually this is a feature request, as you know fat32 systems eat all space
because of clock sizes so a 100Mb tiles folder will eat around 600Mb of
space on the sd card of devices which is odd...

This problem should be fixed but, very important not to sacrifice speed,
the speed of the application is way more important than having less space
on the sd card.

I have done some investigations and found a topic somewhere about truezip
which could solve the problem but will sacrifice speed a lot as the
archives don't support changing contents that easy, what happens sometime
is that a new file is created (you just can't permit creation of another
2GB zip just because you need to add another 3kb image somewhere),
especially on a mobile device.

I saw that truezip is already in lib folder but it's not used. It can't be
used on android as is, so I made a small truezip jar that has only the
essential and don't give errors but still does not work yet. I gave up on
the idea because of the duplicating zip problem.

I propose that in here we should join all infos related to this issue in
search of a good and fast implementation for storing images on sd cards.

Also when time comes and I have a quality patch for this issue, I will need
rights to submit it, but that's another story, for now we have to find a
good solution for the issue.

Original issue reported on code.google.com by mihai.ile@gmail.com on 21 Mar 2010 at 1:08

[GoogleCodeExporter]

Hello again. Well that was fast... I think I made it somehow...
I am using the 1.5 branch

What I did was: as you can' get a 1gb file and hope it will be fast on
read/delete/add I joined all final images into tar files. Tar is good as it 
does not
compress(we are talking about png is not that a naive zip will make it smaller 
anyway).

so instead of having mapnik/zoom/folders/images I now have mapnik/zoom/tar 
files.
Each tar has all the images from the corresponding folder and has the name of 
the
folder plus ".tar"

As an example I used the map from Munich city (all 18 zoom levels). Originally 
I got
~90.000 items inside mapnik folder, after using my method I got ~500 items. 
(andnav
will search/update tar files from folders so it will create the tars on launch 
but it
is slow due to sd card being a class 2), so I made a simple tool to convert
everything so andnav will skip this step.

The tar files now have from 2kb (zoom < 10) to 1.5MB which I think is 
acceptable to
update/retrieve content.

So what about the speed?
I really do not have much time implementing this right now, so speed (code
optimizations, javadoc, ...) was not on priority list, but it goes like this:
Greater the zoom levels, more performance drops (normal). On zoom 18 if I use 
device
wifi into an uncached area (small tar files) is speedy on my hero (aside image
download of course), if I use in a cached area (1.5MB tar) moving back and 
forward on
streets you do notice some lag. Now I do not know if this is due to CPU or just 
the
SD card being slow (I have a default 2gb class2 card and I do believe this is 
the
main reason, maybe a class 6 would improve this, don't have one to test).

I would like to submit a patch so you could test this (maybe in a new branch or 
I
don't know), I actually changed only
OSMMapTileFilesystemCache.java and added Apache's ant.jar to build path which 
provide
the tar tools.

Original comment by mihai.ile@gmail.com on 21 Mar 2010 at 8:18

[GoogleCodeExporter]

Hello, Mihai.  I was thinking of doing the same thing.  May I try your patch? 
ringuette at solarmirror.com .

Other devs, if you're listening ... any thoughts on the performance 
implications of
using a zip archive for the lowermost level?

It would certainly make MapTilePacks more usable for the major use-case of 
people
wanting to use their GPS offline.

Original comment by marcring...@gmail.com on 16 Apr 2010 at 4:53

[GoogleCodeExporter]

This is an issue we're also thinking about at osmdroid:
http://code.google.com/p/osmdroid/issues/detail?id=8

Original comment by ThyMythos@googlemail.com on 17 Apr 2010 at 12:42

[GoogleCodeExporter]

Ok, sorry for the delay, I just created a patch (it also includes ad removal, 
but you
do not have to apply that). I added apache's tar package as java classes 
because I
was thinking it would need changes to make it faster to untar, but I did not 
have the
chance to make changes to it.

The code has some bugs I think, I was only interested to test if it would work 
as tar.
The first time you launch andnav and select map it scans to convert the images 
to tar
files, so it could take a while depending on cache size.
I successfully used this on a 2GB card with complete cache of many cities at 
full
zoom without problems. There is a performance penality at full zoom though. I 
think
that with a little tweak it could work (for example always keep a copy of the 
last
untar file so if you are on full zoom you do not have to untar for every scroll,
would help a lot since the full zoom tar files are the ones with most of the 
images)

This patch was made from the 1.5 branch of andnav.

Oh and by the way I am not responsible if the smartphone melts or something due 
to
this patch :P

Original comment by mihai.ile@gmail.com on 23 Apr 2010 at 9:26

Attachments:

• patch.txt

[GoogleCodeExporter]

Sorry for the upcoming off-topic question, but I don't know where to ask and 
anyway,
I'm totally ignorant of the Android platform...

@mihai.ile & al
The patch just made me notice the AdView stuff. It seems the ads require a 
permanent
connection (and therefore a data plan). Is it the case? If yes, could that be
deactivated when AdFreeState is true?

Original comment by christop...@gmail.com on 4 May 2010 at 12:37

[GoogleCodeExporter]

Has there been any advance here? I meant this is really an issue... even with 
the maximum µSDHC size of 32GB on looses so much...

Original comment by calestyo@gmail.com on 13 Jul 2010 at 7:23

[GoogleCodeExporter]

Like ThyMythos, I'm interested for the sake of osmdroid (and my own project)
But I don't understand when the tar would be created? Continuously updated on 
the phone as the tiles are downloaded?
What if I download a big area myself and put it on the sd card?

Original comment by andpet@gmail.com on 13 Jul 2010 at 9:57

[GoogleCodeExporter]

The way I did it was the tar files would be created on startup, from the 
available tiles in cache. Then updated as you download more tiles.

I also downloaded a big area and place it on the sd card. If you do that with 
the tiles, you will have to wait a long time at first startup as the phone 
creates all the tar files. Or you can create the tar files in the right 
structure before placing them on sd card.

The problem is that my patch is quite slow, and may contain bugs. Never 
programmed for android and never saw andnav code before or have a good 
understanding of how it actually works.

Original comment by mihai.ile@gmail.com on 13 Jul 2010 at 10:15

[GoogleCodeExporter]

Thank you for explaining mihai.ile. "Quite slow" sounds discouraging and it 
won't work well with uploaded maps. I won't try to port your patch at the time 
being but might look more into it later. Thanks anyway.

Original comment by andpet@gmail.com on 14 Jul 2010 at 7:59