Database Storage for tiles

[GoogleCodeExporter]

Implement a storage in sqlite dbs for tiles, almost the same way it
is treated with zips.

The issue with zips is that every entry in the zip file is put in memory
at start, which involves a slow startup and a huge use of memory, which causes 
a crash on some phones when loading a zip file containing 50000 tiles.

On another hand, storing the tiles directly on sdcard is not a good solution 
either, since it wastes much space because of the small size of tile files.

There is an alternative way to store tiles, in a database. The tiles
are stored as blob objects in db, and in order to have fast access to a given 
tile, a unique identifier is computed for each tile, which is used as primary 
key.
After running some tests, this seems to work quite well.

The packager is also modified in order to create the db files. The usage didn't 
change, except it looks at the extension of the output file : if it is a zip 
file it behaves as before, and if it is not, it creates the database.

In order for the new packager to work, you have to make it depend on the 
library sqlitejdbc-v056.jar, found here : http://www.zentus.co/sqlitejdbc/, and 
that I attached to the issue.

I use Eclipse and am not very good at ant, so if someone could modify the 
build.xml for the new library, in order for "ant jar" to work for the packager, 
it would be nice.

Original issue reported on code.google.com by valentin.blot0@gmail.com on 4 Jan 2011 at 12:31

Attachments:

• DataBase.patch
• sqlitejdbc-v056.jar

[GoogleCodeExporter]

Sorry, the patch wasn't complete. Here is the complete patch.

Original comment by valentin.blot0@gmail.com on 4 Jan 2011 at 12:37

Attachments:

• DataBase.patch

[GoogleCodeExporter]

Original comment by neilboyd on 4 Jan 2011 at 9:50

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

The one downside is that we have to include the sqlite jar file which adds some 
size to the distribution. It's not a lot, but it would be nice to not have to 
include it. If there is some way to compile without the sqlite jar and require 
the end-user to add it to their project if they want to use it. If it's not 
present, then the class will just throw an exception at runtime and tell the 
user that it is missing. I don't know if this is possible.

And I'm not trying to deflect usage of this - personally, I think using the DB 
is a good idea assuming there isn't too much of a performance hit - but I worry 
that unless the offline map tools embrace the "db as a storage mechanism", this 
may be just a niche provider that doesn't get widespread use. Again, these are 
just my thoughts - in the end, it's better if we include it than not!

Original comment by kurtzm...@gmail.com on 5 Jan 2011 at 4:02

[GoogleCodeExporter]

I agree with Marc. I was thinking of implementing this patch, but not adding it 
to the default provider chain.

I'm also tending to move away from the tile packager in favour of Mobile Atlas 
Creator.

Original comment by neilboyd on 5 Jan 2011 at 6:32

[GoogleCodeExporter]

I agree. Remark however that the jar file is not included in the osmdroid 
library (the one which is on the phone), but only in the packager jar.

But indeed I would be glad to integrate it as a new output for Mobile Atlas 
Creator. I will try to propose a patch on the sourceforge of the project.

The thing is that without any way to create the db files, the patch is useless 
in osmdroid library, so integrating it to the packager was the first step, 
waiting for this to be integrated in Mobile Atlas Creator.

Anyway, did you try it ? What do you think ?

Original comment by valentin.blot0@gmail.com on 5 Jan 2011 at 11:23

[GoogleCodeExporter]

Please have a look at revision 680. It should now be easier to do this in a 
similar way to ZipFileArchive.

Original comment by neilboyd on 6 Jan 2011 at 11:28

[GoogleCodeExporter]

I've committed the packager parts in revision 682.

Original comment by neilboyd on 6 Jan 2011 at 11:55

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

It was trivial to implement this like I said in comment 6 - done in revision 
683.
I haven't tested this at all.

Original comment by neilboyd on 6 Jan 2011 at 12:08

[GoogleCodeExporter]

Original comment by neilboyd on 6 Jan 2011 at 12:09

• Changed state: Started

[GoogleCodeExporter]

The developer from mobile atlas creator pointed out that the key calculation 
could produce an overflow on high zoom levels, so we should use a "long" 
instead of an "int". Here is the corresponding patch.

Original comment by valentin.blot0@gmail.com on 10 Jan 2011 at 1:11

Attachments:

• long.patch

[GoogleCodeExporter]

Committed in revision 708. Good to see it's getting into Mobile Atlas Creator.

Original comment by neilboyd on 11 Jan 2011 at 9:19

[GoogleCodeExporter]

Done, it's now in Mobile Atlas Creator (in the latest alpha version).

Original comment by valentin.blot0@gmail.com on 15 Jan 2011 at 4:42

[GoogleCodeExporter]

So, does it work? Are we finished? Can I close this now?

Original comment by neilboyd on 15 Jan 2011 at 9:21

[GoogleCodeExporter]

I think so, I didn't make lots of tests but until now it seems to work 
perfectly. The only thing now is to update the wiki page MobileAtlasCreator in 
order to tell that you can also create databases.

Original comment by valentin.blot0@gmail.com on 17 Jan 2011 at 11:34

[GoogleCodeExporter]

Which version of MAC do I need, and what's the name of the Atlas format?

Original comment by neilboyd on 18 Jan 2011 at 9:06

[GoogleCodeExporter]

You need the latest alpha version : 
http://sourceforge.net/projects/trekbuddyatlasc/files/Alpha%20and%20experimental
/MOBAC%201.9%20preview%204.zip/download and you have to chose the "osmdroid 
sqlite" format.

Original comment by valentin.blot0@gmail.com on 18 Jan 2011 at 7:48

[GoogleCodeExporter]

Original comment by neilboyd on 18 Jan 2011 at 8:45

• Changed state: Fixed