Offline maps

[ljbade]

It is a personal goal to have the ability to view maps without needing an Internet connection.

Very useful for activities such as bushwalking our mountain biking.

Plus we know there are others interested in this feature too.

[ljbade]

Some discussion I had with @kkaefer about what was needed for offline maps:

• native core does not yet support support offline maps - on roadmap
• The cache sqlite DB can be reused for storing tiles offline
  • need to add ability to swap out the cache database and set it to a read-only mode
• need a tool that builds an offline database that can be distributed with the app/downloaded
• could also download all individual tiles on first launch, however this is not good strategy
• could be useful to have a way to differentially update only the tiles that have been changed. Like some sort of patch DB, that has higher priority than main DB, and can eventually be somehow merged into the main DB.

First things to work on:

1. enable offline storage in cache DB
2. create a tool to export a map to a cache DB

[incanus]

Removing label & editing title — this isn't specific to Android.

We've accomplished a lot of this in past across various toolsets on both iOS and Android.

• MBTiles is a spec for raster or vector tiles as blobs in SQLite with a known format and metadata table. This is used for prebunding offline maps with an app. It's not designed for adding to later, but could be remedied in some way if this seems like a good idea.
  • https://github.com/mapbox/mapbox-ios-sdk/blob/release/MapView/Map/RMMBTilesSource.h
  • https://github.com/mapbox/mbxmapkit/issues/3
  • https://github.com/mapbox/mapbox-android-sdk/blob/65a1c81f72b3419ae22bc74e1b28adb0a71fbb11/MapboxAndroidSDK/src/main/java/com/mapbox/mapboxsdk/tileprovider/tilesource/MBTilesLayer.java

• need to add ability to swap out the cache database set it to a read-only mode

  • Agreed.

• need a tool that builds an offline database that can be distributed with the app/downloaded

  • https://github.com/mapbox/mbutil

• could also download all individual tiles on first launch, however this is not good strategy

  • It can be a good strategy, depending on usage, and becomes much more viable with vector tiles than raster.
  • https://github.com/mapbox/mbxmapkit/blob/master/MBXMapKit/MBXOfflineMapDownloader.h
  • https://github.com/mapbox/mapbox-ios-sdk/blob/release/MapView/Map/RMTileCache.h#L159
  • https://github.com/mapbox/mapbox-android-sdk/issues/533

[ljbade]

MBTiles is a spec for raster or vector tiles as blobs in SQLite with a known format and metadata table. This is used for prebunding offline maps with an app. It's not designed for adding to later, but could be remedied in some way if this seems like a good idea.

Being able to push out small update datasets will be needed if people start creating maps of whole countries, e.g. topo maps for all of Australia or NZ cause it will be annoying to redownload hundreds of MBs of map data every time the dataset gets updated (NZ updates their topo data monthly)

[ljbade]

It can be a good strategy, depending on usage, and becomes much more viable with vector tiles than raster.

True keep forgetting that vector data is a lot smaller.

[ljbade]

Perhaps as a first step, we could add support for pre-caching a bbox + zoom level of data?

[ljbade]

Some ideas of minimum functionality:

• function to calculate number of tiles needed for given bounding box and zoom range
• function to start offline caching - will take bbox, zoom max & min, some sort of callback/interface to receive progress notifications
• notification that receives - total number of tiles, tiles downloaded so far, number of failures (and possibly list of the failures), overall finished & successful indication

[ljbade]

After a quick look through the code here is a strategy:

• build list of all Sources needed for each layer
• for each Source tell it to fetch all needed TileData
• ensure TileData downloads and saves data into SQLiteStore via FileSource/HTTPRequest
• disable cache expiry check in HTTPRequest
• ???

[ljbade]

We will also need to save in cache:

• Style JSON
• Tile source JSON
• PNGs for styles

Is there anything else needed to be downloaded to render a map?

[ljbade]

Hmm, going to take me a while to figure out how to do this tidily.

Suggestions?

[incanus]

Perhaps as a first step, we could add support for pre-caching a bbox + zoom level of data?

We have built this out pretty robustly on iOS, so that's a good starting spot.

iOS SDK

In RMTileCache:

Estimates

-tileCountForSouthWest:northEast:minZoom:maxZoom:

Start/stop

-beginBackgroundCacheForTileSource:southWest:northEast:minZoom:maxZoom:
-cancelBackgroundCache

Progress

-tileCache:didBeginBackgroundCacheWithCount:forTileSource:
-tileCache:didBackgroundCacheTile:withIndex:ofTotalTileCount:
-tileCacheDidFinishBackgroundCache:
-tileCacheDidCancelBackgroundCache:

Storage format is a SQLite-based common store keyed on tile source, per the convention of the upstream project that we built this SDK off of. This same cache is used for performance caching during online use as well as offline fallback.

MBXMapKit

In MBXOfflineMapDownloader:

Start/stop

-beginDownloadingMapID:mapRegion:minimumZ:maximumZ:
-cancel
-resume
-suspend

Estimates & Progress

-offlineMapDownloader:stateChangedTo:
-offlineMapDownloader:totalFilesExpectedToWrite:
-offlineMapDownloader:totalFilesWritten:totalFilesExpectedToWrite:
-offlineMapDownloader:didEncounterRecoverableError:
-offlineMapDownloader:didCompleteOfflineMapDatabase:withError:

Here we had a combination of a clean slate for how to do this along with integration with an existing Apple API. Storage format is again SQLite but offline maps are treated as atomic, immutable documents.

We don't necessarily want to mirror the APIs to Java for any sort of undue parity, but these are some ideas. I think the idea of bounding box + zoom range, start/pause/cancel, and progress callbacks are the general model we want.

However, we should also decide whether to do this at the C++ level in some core way.

In my eyes, despite this being eventually important, it's far lower priority than general Android parity with current iOS tools + buildout of client side data (https://github.com/mapbox/mapbox-gl-native/issues/507) so that mobile apps can start deploying on GL in the wild.

[hallahan]

Whatever happened with this issue? Is there now MBTiles support for Android? Is this done natively? This feature is a requirement for me to use this library as an Android app.

[incanus]

There is not MBTiles support yet in this project. Offline support will be coming in a future release; pretty sure this will include MBTiles.

[hallahan]

Darn, is there a branch this is being tried with? Is this going to be native C++ for all builds? For me, this is the most important feature I need. I'm interested in helping.

[incanus]

There's no movement on it yet, AFAIK. Right now we are sprinting towards a series of iOS beta releases, after which we'll focus on Android and a broader feature set. I would expect it to be in C++, yes, with Java/ObjC bindings.

Happy to take work in a branch, but it just likely won't get many eyeballs during March due to priorities elsewhere.

[kristfal]

Hey,

I know this is a large feature that will be implemented later, but I'm very curious regd. the performance differences, so wondering:

Is there currently a more elegant way to load tiles bundled with an app besides doing this and then load individual .pbfs referenced through a tilejson?

[xwz]

I've made some progress in adding mbtiles source. It is very basic at the moment, it simply loads tiles from the mbtiles file using sqlite.

[ljbade]

@xwz Great, have you put your changes on GitHub?

[xwz]

@ljbade See https://github.com/mapbox/mapbox-gl-native/compare/master...xwz:mbtiles-source

[vespakoen]

Great stuff, @xwz what is the reason that there is no pull request for your changes?

[nitrag]

Bump/Vote! Want!

[erf]

Same her! Please support this in Android.

[PaulEhrhardt]

Yes, me too. Please add offline support.

[erf]

What we need is to be able to point a style.json tiles source to a local file path for hosting .pbfs, fonts and sprites.

[frederik-jacques]

This would be a great addon, we are developing citytour apps with Mapbox & local mbtile files. It would be great to see a possibility to add local vector maps (smaller size then mbtile files), because our users don't need internet connectivity when they do a tour (most are on wifi only ipads)!

[chrisballinger]

We use offline maps (mbtiles) in iBurn because there's no internet in the Black Rock Desert: https://github.com/burning-man-earth/iburn-ios

Super excited to add vector maps next year!

[incanus]

I've started on some work to abstract a per-platform tile request hook over in platform-tile-hook, meaning iOS or Android could jump in with reads out of SQLite or what-have-you, eventually on a per-frame basis (for things like frames of video https://github.com/mapbox/mapbox-gl-native/issues/601 or weather rasters). I'm not in that work right now, but we should be picking it up again soon.

[mb12]

@incanus Wouldn't this make it difficult for Mapbox to charge based on number of MapViews/metered usage etc? Also would this mean that metrics collection happen offline and periodically synced to mapbox?

[incanus]

The business questions are further out, particularly because we promise no longer than 60s live time for metrics data on device before it's just discarded as part of our TOS. But this needs to happen on a technical level.

[chrisballinger]

@incanus Oh hmm, would this mean we wouldn't be able to continue using the open source / free tier for offline vector maps in iBurn?

[incanus]

I don’t think we’re sure yet. We will be digging into this more once we pick up the offline development again as a priority.

[chrisballinger]

@incanus Please consider a free tier for verified open source projects! We operate on a $0 budget and really love what you guys are doing. Because you guys understand the value of open source tech, I've always recommended Mapbox to others, and have no problem with paying for commercial use. :)

[incanus]

@chrisballinger Duly noted :smile:

[ltressens]

After reading all these comments, and my small knowledge of all distinct branches in this project, I have no clear idea if the Offline feature for Mapbox GL iOS is nearly ready, or if it has a public schedule. So I welcome every answer to my comment :) Thanks

[tyjacobs]

I too am actively working on an app for a client that needs offline maps. The approach of having each user's device cache tiles while connected and then having them available offline is fine in our case. That is, assuming we can programmatically cache on a background thread similar to what was mentioned above:

-beginDownloadingMapID:mapRegion:minimumZ:maximumZ:

For now, my project only needs to support iOS. I am in the same boat as @ltressens, wondering if/when this capability will be available.

[nitrag]

+1 Don't really care about the GL portion, need offline (mbtiles) file capability!

On Mon, Aug 31, 2015 at 11:29 AM, Ty Jacobs notifications@github.com wrote:

I too am actively working on an app for a client that needs offline maps. The approach of having each user's device cache tiles while connected and then having them available offline is fine in our case. That is, assuming we can programmatically cache on a background thread similar to what was mentioned above:

-beginDownloadingMapID:mapRegion:minimumZ:maximumZ:

For now, my project only needs to support iOS. I am in the same boat as @ltressens https://github.com/ltressens, wondering if/when this capability will be available.

— Reply to this email directly or view it on GitHub https://github.com/mapbox/mapbox-gl-native/issues/584#issuecomment-136406293 .

[ltressens]

Well, understand me well @nitrag. I DO care a lot about Mapbox GL. I am already a user of Mapbox iOS SDK which allows caching for offline use (you should be happy). My interest is in caching vector information for Mapbox GL.

[nitrag]

Oops. I was speaking to needing offline support on the Android SDK. Maybe that's a different Issue #. On Sep 1, 2015 4:11 AM, "ltressens" notifications@github.com wrote:

Well, understand me well @nitrag https://github.com/nitrag. I DO care a lot about Mapbox GL. I am already a user of Mapbox iOS SDK which allows caching for offline use (you should be happy). My interest is in caching vector information for Mapbox GL.

— Reply to this email directly or view it on GitHub https://github.com/mapbox/mapbox-gl-native/issues/584#issuecomment-136628828 .

[leimingdu]

We are also developing an offline man using vector title, need cache all tiles for both iOS and android, any progress? whether it is possible to using sqlite DB? thanks

[incanus]

No recent movement on this.

[ltressens]

:( Still on schedule or out of scope for a moment ? Le ven. 11 sept. 2015 à 18:30, Justin R. Miller notifications@github.com a écrit :

No recent movement on this.

— Reply to this email directly or view it on GitHub https://github.com/mapbox/mapbox-gl-native/issues/584#issuecomment-139592169 .

[AlanPew]

In addition to offline tiles, I've noticed that vector tiles cannot be used when setting the url similar to how the raster tiles can be set in the style.json file from an outside url. Specifically, I've set the style.json source to use third party vector tiles similar to this Mapbox GL JS example using Mapzen vector tiles (https://www.mapbox.com/mapbox-gl-js/example/third-party/). I've set up the source and style of the layer exactly like the third party source styling for Mapbox GL JS and the feature isn't available. Are third party vector tiles going to be able to used as a GL layer, or are vector tiles only going to be allowed from a Mapbox url? This might seem like a tangent from the original question, but I'm posting here because its of similar relevance to using vector mbtiles hosted on the iOS device (cannot use local mbtiles, only ones stored on Mapbox servers). Is this going to be a feature similar to how raster tiles can use third party source, or is it not a priority right now? Love how great the WebGL renders and looks and would love to see this availble. Thanks for your hard work, @incanus .

[jfirebaugh]

@AlanPew That should work. It's really a different issue -- can you open a separate ticket please?

[AlanPew]

Will do @jfirebaugh. Thanks for the heads up and sorry for straying from the original issue everybody.

[caBBAlainB]

+1 to get offline MB for Android.

[kkaefer]

@erf you can also just put them directly on disk and specify with asset://tiles/{z}/{x}/{y}.pbf

[ljbade]

@erf Yeah has to be in assets folder. However it is possible to change that behaviour in C++.

[suraj-aahlaad]

please any one tell me How to calculate the selected area of the map size before downloading the map.

[twbell]

A quick heads-up here to note that we believe offline operation is critical to the future of mobile mapping, and we are therefore baking this capability at a low-level into both our iOS and Android clients. We intend our implementation to meet requirements for both performance (obviating bad network connections) and offline travel/planning ("download this area before I go there") . We will begin posting updates shortly to this ticket as we make progress.

[tikimcfee]

Hey there!

I'd like to put in the similar +1 from @hallahan, @erf, @leimingdu, et al. MapBox is a proven and core asset in the production of our team's software solution. We cache tiles for offline use in iOS to great success, and have been pushing to implement the same feature set on Android. One of our biggest gating factors for this, though, is our clarity in what we can expect this GL push to mean for both mobile platforms in terms of overall client capability and release schedules. We would hate to spend parity-focused development time for an SDK right on the cusp of major changes!

Our critical scenarios and questions are, then, focused on:

1. Our mobile clients will move offline and online frequently - having map tiles available for users at all times is critical to our user experience around that scenario. To that end, we need to know if both Android and iOS will have the capability of caching tiles (raster or vector).
2. We are an agile team with limited development time, and rework being necessitated from staggered platform SDK releases are brutal for us. Will the SDKs be ready for both platforms at or around the same time? If not, is there an expected release schedule that we can keep in mind?

I think those listening in here would really appreciate a consolidated answer on this (I know my team will!). We also want to put out a hand for any sort of questions you might have for us on this, and another for any help we can lend in for testing or describing use cases in greater detail.

Thanks for the time on this, and again for picking up this feature with gusto!

[twbell]

Copy that @tikimcfee.

Our approach on the Mobile side here is for broad feature parity between iOS and Android. We expect this frequently to be a stepping-stone approach where one platform leads and the other follows in short order -- which leads first will vary depending on internal business priorities, but the goal is to minimize feature lag between the two platforms.

We want our mobile dev intentions to be lucid and reliable. To this end we are keen to articulate our priorities and our concomitant release schedule wherever possible. The two caveats here are (1) we are starting very conservatively, and (2) we can expect to have internal business drivers that may divert resources on short notice.

That being said, I would hope to continue to make our intentions more plain and our plans more specific as we continue to develop our mobile clients.