Use transparent zlib library for compressed OsmAnd maps

[hvdwolf]

Currently the OsmAnd files are big (not to say huge). Uncompressed some almost run into the (old) 2GB FAT-16 limit. (And I know this has been led to the decision to split countries in smaller size due to this limit) We are now currently using the OBF version 2 format (since 2011?). Please create an OBF version 3 map format based on zlib compressed maps. The zlib license is a "free software license" comparable with GPL.

zlib and compressed maps have the following advantages:

• using zlib: (very) fast decompressing using low system resources.
• compresses very well (zlib -> DEFLATE equal to zip).
• Much faster read times of maps due to ~40-60% maps size and transparent decompression, possibly also leading to (much?) faster rendering on screen
• No need to add extra zip compression step after map creation as the maps are already compressed (same as normal zip compression which also uses the DEFLATE algorithm)
• No change in bandwidth download times as the zlib/gzip DEFLATE compression is the same as the default ZIP DEFLATE compression routine.
• Even slightly smaller files than zip files ( and gzip files) because the zlib header is smaller than that of zip (and gzip), although that won't be noticable on 200MB+ size maps.

"Irrelevant" counter arguments: :wink:

• Storage is cheap, so why compress the maps? This is not a valid argument as it will stop us from improving the map structure and application performance. Yes, storage is cheap, but a compressed transparent format might increase reading speed enormously.
• Incompatibilty with older versions/formats. Give it a different extension (suffix) or extra identifier before the suffix. Older OsmAnd versions will simply read .OBF files. Newer versions can read .OBF and *.OBF3 (as naming example). Or with identifier Netherlands_europe.zlib.obf or Netherlands_europe.road.zlib.obf, equally or comparable like Netherlands_europe.road.obf or Netherlands_europe.wiki.obf

[hvdwolf]

Note: If the obf format is heavily sqlite(3) based, one can still use the zlib library. Sqlite knows the CEROD(*) extension for compressed databases also based on the zlib library.

*: CEROD: Compressed and Encrypted Read-Only Database

[vshcherb]

I don't think it's actually very relevant but we can't afford zipping maps cause binary references inside the files will be broken, so indexes won't work correctly. You need to zip atomic bulks of data, so you can access and build indexes on top of it.

[hvdwolf]

I don't have the background to judge that, but I do know other navigation apps using the same technique. Considering the fact that zlib is transparent as a layer between the reading logic and the data, it should not matter. If it's not feasible, then so be it. But has it actually ever been examined?

[vshcherb]

No, but it's not clear the ultimate goal of it. Making maps 20% smaller doesn't look very attractive to a deep research and refactoring. So the truth is to make some incremental steps to improve situation but I don't know which steps exactly and how to make yet.

[hvdwolf]

Not 20% smaller. It is the same as the currently zip compressed map. Both zip and zlib use DEFLATE. So compressed map size will be 40-60% of that of an uncompressed map. Like said: size is not the goal. Improving reading speed is the goal, thereby hopefully improving rendering speed as well.

I can't do it myself so I will leave it with this. Hopefully some time in the future. You can close after reading :)

[trou]

For what it's worth, zstd is probably better than zlib: faster and more efficient.

[hvdwolf]

Great 😃 . I fully agree to implement that. But I don't think it will be implemented in the near future.

[tiziodcaio]

First of all, how maps are generated? We should add a new workflow which generates compressed maps, then implement the support in osmand. It should not be such that hard to do!

[sonora]

In my experience with recent optimization work (on our gpx file handling), OsmAnd spends a significant amount of time on I/O.

Chances are, as @hvdwolf already noted above, supporting to work with files zipped in storage could actually make the app snappier by reducing I/O. And this may be true for all types of files, gpx and maps alike.

Would in deed be interesting to test...