search

mapbox / mbxmapkit

DEPRECATED - Lightweight Mapbox integration with MapKit on iOS
http://mapbox.com/mbxmapkit
Other
335 stars 85 forks source link

Discuss API refinements, including caching #73

Closed ghost closed 10 years ago

ghost commented 10 years ago

Over at issue #61 we considered a refactored design for asynchronously loading TileJSON, and at pull request #71 we discussed my issue61 branch with a new approach to solving problems around the asynchronous TileJSON stuff.

The conclusion from #71 seems to be that the general approach in the issue61 branch is sound, but it needs to be packaged within an API which feels consistent with Apple's API's in general, and MKMapKit in particular. So this ticket is for discussing refinements to the API. One of the big open questions is what the caching will look like.

My plan for the moment is:

I'm guessing this iteration will reach a good stopping point by approximately March 20th (next Thursday).

ghost commented 10 years ago

The new branch for working on this is at mapbox:issue73

ghost commented 10 years ago

Something about upgrading to Xcode 5.1 yesterday seems to have messed up the maps entitlement for the new OS X demo app (the old MBXMapKit 0.1.0 demo app still works though). Probably this is just a local provisioning profile issue on my development box, but it may pop up elsewhere.

ghost commented 10 years ago

The Xcode provisioning profile thing for the OS X demo seems to be fine now. I did end up committing some changes to the project file as part of getting it working again.

ghost commented 10 years ago

Based on my experimenting so far, it doesn't seem to be possible to conveniently determine when the data from an NSURLSession data session task came from cache (vs an HTTP download) just by looking at the response. That's inconvenient, because I want to gather statistics about cache hits and misses in order to determine if, and how well, caching strategies are working.

The one option I can see so far is to implement URLSession:dataTask:willCacheResponse:completionHandler: from the NSURLSessionDataDelegate Protocol and have it add a userInfo dictionary with an entry flagging the response as having been locally cached. The responses do have caching related headers, but those are all about what's happening server-side and at the HTTP level. I want to be able to tell the difference between a freshly downloaded tile which was a CloudFront cache hit and the same tile (with its associated same response!) which might come out of the local disk cache 20 minutes, or 2 days, later.

ghost commented 10 years ago

The NSURLSessionDataDelegate idea didn't work because, as Apple's documentation explains, "If you create a task using a method that takes a completion handler block, the delegate methods for response and data delivery are not called."

ghost commented 10 years ago

I'm hitting a world of pain as I try to verify what NSURLCache + NSURLSession are actually doing in response to my requested cache and session configurations. My suspicion is that none of my NSURLSession requests are actually getting served from cache, and instead they are always resulting in new HTTP requests. The thing that concerns me is that, as far as I can tell, all the responses I'm getting are NSURLHTTPResponse, but some of them ought to be NSCachedURLResponse.

In order to get more visibility into, and control over, what happens during network requests, it looks like I might need to stop using data tasks with completion handlers and come up with an alternate approach based on implementing the NSURLSessionDelegate, NSURLSessionTaskDelegate, and NSURLSessionDataDelegate protocols.

ghost commented 10 years ago

I figured out a way to verify that tiles and JSON are definitely getting written to cache, at least some of the time, but I still haven't found a way to tell if and when subsequent requests are getting served from cache.

On my iPhone, the current cache configuration creates a Cache.db sqlite database in the app's sandbox at /AppData/Library/Caches/com.mapbox.MBXMapKit-iOS-Demo-v030. If you open that up and do a sqlite> SELECT request_key FROM cfurl_cache_response;, it dumps out a list of all the urls in the cache (lots of pngs for each map, along with metadata and marker json). So, I guess that's progress.

ghost commented 10 years ago

I've been running my phone's traffic through Charles Proxy and experimenting with setting cache policies like this:

NSURLRequest *request;
NSURLSessionDataTask *task;
request = [NSURLRequest requestWithURL:url
           cachePolicy:NSURLRequestReturnCacheDataDontLoad timeoutInterval:60];
task = [_dataSession dataTaskWithRequest:request completionHandler:^(NSData *data, 
        NSURLResponse *response, NSError *error){ /* ... */ }];

My conclusions are:

  1. The caching is definitely working, and lots of images/json are getting returned from cache.
  2. With the default cache policy, some of the cache hits are re-validated, so if you're watching traffic in a proxy, but not paying close attention to the headers and response codes, it might look like the resources are being re-downloaded.
    Revalidation requests for local cache hits include If-None-Match: ... and If-Modified-Since: ... headers, and regular requests for local cache misses don't include those headers. Responses can look like X-Cache: RefreshHit from cloudfront, X-Cache: Hit from cloudfront, and X-Cache: Miss from cloudfront. Presumably other X-Cache responses might also be possible.
  3. My existing notification mechanism for HTTP success, HTTP failure, and network failure is currently counting local cache hits as successful HTTP requests (that's bad). I'm not sure if it's possible to distinguish between a response that comes from a local cache hit and a response that just came over HTTP. I'm wondering if the thing to do might just be abandon my attempts to measure cache stuff from inside the app and instead just rely on an external proxy to make sure the cache is actually working.
incanus commented 10 years ago

Responses can look like X-Cache: RefreshHit from cloudfront, X-Cache: Hit from cloudfront, and X-Cache: Miss from cloudfront.

Just in case it's not clear, this has to do with whether tiles are missing and going all the way back to the backend render server, or if they are hitting and cached as rasters in the CDN.

ghost commented 10 years ago

I'm pretty close to convinced it's not possible to distinguish between an NSURLHTTPResponse that comes from a local cache hit and one that comes fresh off an HTTP connection. A bit ago I wrote,

The thing that concerns me is that, as far as I can tell, all the responses I'm getting are NSURLHTTPResponse, but some of them ought to be NSCachedURLResponse.

and now I'm pretty sure I misunderstood the purpose of NSCachedURLResponse. As far as I can tell, it is only used with - (NSCachedURLResponse *)cachedResponseForRequest:(NSURLRequest *)request and - (void)storeCachedResponse:(NSCachedURLResponse *)cachedResponse forRequest:(NSURLRequest *)request from NSURLCache.

After some time in the debugger looking at response objects coming back from dataTaskWithRequest:completionHandler:, I'm pretty convinced that there isn't any way to tell when you're getting a cached data/response pair. It is possible to tell when something is getting written to cache, but I can't see any way to tell when data is coming back out of the cache. NSURLCache seems to do its job well based on the caching policy you set, but it apparently doesn't like to share a lot of details about what's going on under the hood. This seems like a potential Radar.

ghost commented 10 years ago

@incanus re caching headers... yeah, all the server side stuff is fairly straightforward. I've been searching for any hints of about whether my responses are hits or misses from the local cache, and I'm just using this issue to take some notes.

ghost commented 10 years ago

Note to self: for persistent offline tile storage, Justin's recommended approach is: don't impose any arbitrary upper limit on the number of tiles it will attempt to download, and do provide a delegate callback which gives progress updates on percent of total completion.

Also, I should take a look at RMTileCache and its delegate in the sdk.

ghost commented 10 years ago

As I'm getting ready to take a swing at background tile downloading for offline maps, I see there's been a fair amount of relevant discussion over in the iOS SDK issues:

ghost commented 10 years ago

Apple's recommendations about directories for storing app data and how to exclude files from iCloud backups:

ghost commented 10 years ago

Some notes on tile addressing schemes:

ghost commented 10 years ago

RMTileCache includes these things,

- (void)beginBackgroundCacheForTileSource:(id <RMTileSource>)tileSource
                                southWest:(CLLocationCoordinate2D)southWest
                                northEast:(CLLocationCoordinate2D)northEast
                                  minZoom:(float)minZoom
                                  maxZoom:(float)maxZoom;

- (void)cancelBackgroundCache;

and RMTileCacheBackgroundDelegate includes these,

- (void)tileCache:(RMTileCache *)tileCache
            didBeginBackgroundCacheWithCount:(int)tileCount
                               forTileSource:(id <RMTileSource>)tileSource;

- (void)tileCache:(RMTileCache *)tileCache
            didBackgroundCacheTile:(RMTile)tile
                         withIndex:(int)tileIndex
                  ofTotalTileCount:(int)totalTileCount;

- (void)tileCacheDidFinishBackgroundCache:(RMTileCache *)tileCache;

- (void)tileCacheDidCancelBackgroundCache:(RMTileCache *)tileCache;

That stuff seems like a good starting point for how the MBXMapKit API should present its functionality for persistent storage of offline maps. However, I want to avoid using the word "cache" because the conceptual distinction between cache and persistent storage is really important here. For all the use cases of offline maps which I've ever heard about, it's really bad if tiles that you've previously downloaded disappear while you're away from a network connection. Basically tiles that you've downloaded need to remain available indefinitely until you specifically decide to get rid of them.

One of the key ideas of caching is that, by design, things stored in the cache are expected to sometimes be deleted or replaced behind the scenes for performance reasons. Those are the wrong semantics for offline map tile storage. What's needed is more like a filesystem or a database, where the expectation is that things stay where you put them.

incanus commented 10 years ago

I want to avoid using the word "cache" because the conceptual distinction between cache and persistent storage is really important here.

:+1:

ghost commented 10 years ago

I'm thinking something like this might be good:

@interface MBXOfflineTileCollection : NSObject

+ (void)beginBackgroundDownloadOfMapID:(NSString *)mapID 
                             southWest:(CLLocationCoordinate2D)southWest
                             northEast:(CLLocationCoordinate2D)northEast
                               minZoom:(float)minZoom
                               maxZoom:(float)maxZoom;

+ (void)cancelBackgroundDownloadOfMapID:(NSString *)mapID;

@end

Perhaps in combination with something like this:

@interface MBXRasterTileOverlay : MKTileOverlay
...
- (id)initWithOfflineTileCollection:(MBXOfflineTileCollection *)tileCollection;
...
@end
incanus commented 10 years ago

Since we're MapKit-specific, I'd recommend just using an MKCoordinateRegion instead of direct CLLocationCoordinates as well as NSInteger for the zoom levels a là -[MKTileOverlay minimumZ] but yeah, this looks good. Making it mapID-specific unlike the SDK is ok due to the lesser scope of this project.

ghost commented 10 years ago

Okay, so more like this:

@interface MBXOfflineTileDatabase : NSObject

+ (void)beginBackgroundDownloadOfMapID:(NSString *)mapID 
                      offlineMapRegion:(MKCoordinateRegion)offlineMapRegion
                              minimumZ:(NSInteger)minimumZ
                              maximumZ:(NSInteger)maximumZ;

+ (void)cancelBackgroundDownloadOfMapID:(NSString *)mapID;

@end

with this:

@interface MBXRasterTileOverlay : MKTileOverlay
...
- (id)initWithOfflineTileDatabase:(MBXOfflineTileDatabase *)tileDatabase;
...
@end

[edit: I changed "Collection" to "Database" because this isn't going to work anything at all like stuff involving UICollectionView]

ghost commented 10 years ago

Another thing to take a look at re: estimating the number of tiles needed for a background tile download: https://github.com/mapbox/mapbox-ios-sdk/issues/172

ghost commented 10 years ago

@incanus per chat earlier, can you take a look at MBXBatchDownloadTask.h and MBXOfflineMapDatabase.h from the issue73 branch?  I'm trying to make something that feels like the cancel/suspend/resume stuff from NSURLSessionDownloadTask, but which reflects the fact that the unit of work is a map region (many small related files) rather than a URL (one big file)

incanus commented 10 years ago

Cool. Some notes:

Overall

Since we're getting into task and manager territory, and we've pretty much decided to grow in file count, does it maybe make sense to leverage AFNetworking for some of the work? A downside here would be namespace collision problems with apps using both MBXMapKit and AF, though.

MBXBatchDownloadTask.h

MBXOfflineMapDatabase.h

ghost commented 10 years ago

@incanus Hmm... interesting. Your comments give me the impression that I need to run through this all again and think through a couple things a bit more carefully. For instance

how long do we keep these completed tasks around? What is the cleanup mechanism?

Omitting something like + (void)deleteTaskForMapID:(NSString *)mapID; was an oversight. I have a mechanism in mind for that (a folder full of mbtiles sqlite db's), but my planning around that still feels kind of half-baked. Also, I'm not convinced that trying to mimic NSURLSessionDownload task is the right way to manage batch downloads and the creation of MBXOfflineMapDatabase objects. This area needs some more thought.

On the other hand, when it comes to the MBXOfflineMapDatabase part, I have an implementation in mind which I feel pretty confident about. Basically I want MBXRasterTileOverlay to have an initializer like (id)initWithOfflineTileCollection:(MBXOfflineTileCollection *)tileCollection; - (id)initWithOfflineMapDatabase:(MBXOfflineMapDatabase *)mapDatabase; which configures the overlay to pull URLs out of an sqlite database configured as a key value store.

For the -dataForKey:withError: method, this is tile image (or in future, vector) data, yes?

What I'm envisioning is that all files needed to render an offline map region would be stored as blob values with URLs as the keys. Since I've arranged MBXRasterTileOverlay so that all of its HTTP requests go through a single function, it would be pretty straightforward to add a branch there based on whether the overlay layer had been initialized with an MBXOfflineMapDatabase. In the offline case, it would just call -dataForKey:withError: with the URL instead of requesting the URL through NSURLSession.

On the file system, I'm expecting this to look like a folder in Application Support or someplace, with the flag set to exclude its contents from iCloud backups, where there would be an sqlite database for each partial or complete batch download.

Shifting topics slightly,

The -taskForMapID: implies there will be only one, so this reinforces the decision we made to not do multiple simultaneous regions per map ID, right?

Yes, sort of, but I would place the emphasis differently. The key idea from that thread is that we need to clear the hurdle of being able to do an atomic update for one mapID (i.e. keep the old offline map around until we've completely finished downloading a current one to replace it with). We talked about only one map per mapID in the context of an implicit assumption that implementing something which supported multiple regions per mapID would be substantially harder/worse/more-confusing.

However, I'm beginning to think it might actually be much more straightforward to build something which didn't try to police how many offline map databases (referring more to the sqlite files on disk rather than instantiated MBXOfflineMapDatabase objects) you want to keep laying around per map. The thing which would keep it all sane is having a 1 to 1 binding between a particular offline database and the MBXRasterTileOverlay which it was used to initialize. Nothing about that suggests any compelling reason to prevent people from making multiple raster overlay layers backed by different offline map databases.

One other constraint we haven't really talked about, and I think this is actually the one which will prove to be the the most influential in shaping implementation, is the need to handle background downloading well. Suppose somebody downloads 300MB worth of tiles and they're 5MB away from completing a map, then their app gets killed due to memory pressure [edit: or their battery dies, or they have to get on a plane, or whatever]... a) we'd better not lose that 300MB of progress, and b) we need to facilitate starting and stopping cleanly as there are many error handling situations which come up with background tasks.

On a somewhat related note,

does it maybe make sense to leverage AFNetworking for some of the work?

I'm generally unfamiliar with AFNetworking other than having spent some time earlier this evening looking through the documentation. My perhaps under-informed impression is that I don't see any obvious advantages to using it in this situation. If you have something specific in mind, I'm all ears.

ghost commented 10 years ago

On second thought,

One other constraint ... is the need to handle background downloading well.

It might be very reasonable to just take a position along the lines of, "If you want to download offline maps, we strongly suggest that you go some place with good wifi and power, plug in your device, turn off auto-lock, and leave your app open to the download progress indicator screen".

By taking that approach we could a) configure an NSURLSession to run full throttle with multiple simultaneous downloads, and b) avoid substantial hassles associated with background downloads (i.e. constantly queueing up download tasks, copying the resulting temporary files into an sqlite db, and handling many special error situations). We could just suspend the download any time the app became inactive and provide a reliable mechanism to (at the developer's discretion) resume when the app becomes active again.

incanus commented 10 years ago

Re: the first set of comments:

stored as blob values with URLs as the keys

I would probably advise against that, since tile URLs can and will change. I'm thinking about the non-tile specific stuff -- we don't want to bind too tightly too that, plus there are HTTPS and other redirects involved. It's probably safer to stick with the tile z/x/y and mapID combo as the unique identifier per map tile.

The thing which would keep it all sane is having a 1 to 1 binding between a particular offline database and the MBXRasterTileOverlay

Maybe NSUUID use so that particular data sets can be referenced in code, property lists, state preferences, and the like?

We already discussed AFNetworking and the choice of SQLite offline, but to summarize: nothing in particularly with AF; just a suggestion and it sounds like you've thought through both the use of SQLite as well as keeping things distinct from the MBTiles schema for this use.

Re: the second comment:

If you want to download offline maps, we strongly suggest that you go some place with good wifi and power, plug in your device, turn off auto-lock, and leave your app open to the download progress indicator screen.

I don't think we can limit to this use case. The idea of "background map downloads for offline use later" is pretty broad and frankly a major selling point in our system, I think. I think we should design for the "anytime" use, then down the road possibly consider detecting and taking advantage of situations where resource hogging is permitted and desirable.

we need to facilitate starting and stopping cleanly as there are many error handling situations which come up with background tasks.

Goes in hand in hand with above -- this is always the case for mobile APIs and we should probably design for the crappiest of situations and make a solution that works well.

ghost commented 10 years ago

we should probably design of the crappiest of situations and make a solution that works well.

Hmm... okay. I've been thinking about this from the perspective of "What's the simplest and most straightforward way of enabling offline maps, since right now we haven't got anything?", but it sounds like you feel there's a worthwhile payoff for aiming considerably higher with our first pass at this.

That works for me, it just means that instead of 1x effort to make it happen, it will be more like 2x or 3x. I agree that clearing a higher bar would be cool.

Maybe NSUUID use so that particular data sets can be referenced in code, property lists, state preferences, and the like?

No, I don't think anything like that is necessary. What I meant by "1 to 1 binding" is that trying to have one MBXRasterTileOverlay instance pull data from multiple MBXOfflineMapDatabase instances will add an annoying amount of implementation complexity. What I mean is I like the idea of an initializer like - (id)initWithOfflineMapDatabase:(MBXOfflineMapDatabase *)mapDatabase;, but I don't want to go down the road of - (id)initWithOfflineMapDatabases:(NSArray *)mapDatabases; If people want multiple offline database layers on their map, they can just make more than one MBXRasterTileOverlay instance.

ghost commented 10 years ago

I would probably advise against [stored as blob values with URLs as the keys], since tile URLs can and will change. I'm thinking about the non-tile specific stuff -- we don't want to bind too tightly too that, plus there are HTTPS and other redirects involved. It's probably safer to stick with the tile z/x/y and mapID combo as the unique identifier per map tile.

Hmm... I'm not sure I buy the part about "URLs can and will change". Right now MBXMapKit uses hardcoded URL templates for tiles, TileJSON, marker geoJSON, and marker icons. What I'm proposing would use those same url templates template generated URLs as keys. In order for those URLs to break, the v3 API would have to be changed or turned off. Are you saying you expect that to happen?

If you're talking about MBXMapKit will eventually need to support the v4 API, that's a different issue, and it can be addressed by updating the URL templates across the board.

Can you expand on how you see the URL key blob value thing being a problem?

incanus commented 10 years ago

On the design front, I just mean we should design first for the use case of a normal, intermittent app, not a remains-unlocked-while-on-wifi-mode special cases app. If we're really feeling ambitious, use of a map view at the same time a background download is happening (which many, many people try with the SDK now).

Re: multiples, makes sense.

incanus commented 10 years ago

On the URLs changing and URLs as keys, it's more the idea of redirects happening, say on a (possibly future permissible) mapID change/redirect in our services, or HTTP -> HTTPS redirects, or other adjustments to the URL. I'm just thinking of typical HTTP URL behavior and keeping it simple -- why tie to URL when the unique bits are the tile triad address and a map ID?

ghost commented 10 years ago

when the unique bits are the tile triad address and a map ID?

Because I'm not just thinking about tiles. I want a map database to include all the resources necessary to load a map: TileJSON, geoJSON, marker icons, and tiles

it's more the idea of redirects happening

That wouldn't cause a problem because redirects don't have a retroactive effect on the URL requests. NSURLSession abstracts all of that by automatically following redirects.

ghost commented 10 years ago

To expand on my last comment a bit, I think we have a different understanding of what the purpose of MBXOfflineMapDatabase would be. I'm not thinking of an MBXOfflineMapDatabase as merely a source of tiles. Rather I want a mechanism which can provide an offline version of all the network resources which are necessary to load a map. That's why I want to use URL requests as the interface point.

NSURLSession takes HTTP URLs and provides NSData responses, and I want to build a mechanism which does the same thing, but without making network requests. That way it will be really easy to hook the network request code to make an offline request instead of an online request.

ghost commented 10 years ago

Skipping back a bit...

On the design front, I just mean we should design first for the use case of a normal, intermittent app, not a remains-unlocked-while-on-wifi-mode special cases app.

Sounds good to me. That should be an interesting challenge.

incanus commented 10 years ago

I want a mechanism which can provide an offline version of all the network resources which are necessary to load a map.

Ahhh, this makes a lot more sense now. Sorry. Ok, this sounds like a good direction to head, then.

:+1:

ghost commented 10 years ago

I just committed yet another take on how the API could work for managing downloads of offline maps.

The main thing I've been thinking about is how to structure stuff so it's possible to preserve state for a download in progress when the app terminates and re-launches. I got rid of MBXBatchDownloadTask which was based on the concept that there could be multiple download tasks in progress, and I replaced it with an MBXOfflineMapDownloader singleton which will only support one background download at a time (canceling deletes the temporary sqlite db, and finishing normally moves the db to a different directory).

My thinking is that there will be, at most, 1 background download job in progress. When an active download job finishes, the download manager will reset back to an idle state, and the temporary sqlite database it was using will be moved into a folder which is managed by MBXOfflineMapDatabase (that part hasn't been filled out yet though).

The big reason this change seemed like a good idea was my desire to avoid:

  1. Exposing a lot of implementation details in header files (e.g. when download manager instantiates objects of a different class to store temporary data), and
  2. Having to figure out how to set the delegates for a bunch of download task objects that are getting restored from disk when the app re-starts.

Said another way, the public API can be a lot simpler if the class which provides cancel/resume/suspend functionality for download jobs manages its own temporary data (i.e. an sqlite db full of downloaded files). Imposing the restriction that only one background download job can be in progress any given time further simplifies things.

ghost commented 10 years ago

Something which will need to be figured out... what is an appropriate Mac OS equivalent to background downloading on iOS? On a related note, how much of the iOS background downloading support naturally belongs in MBXMapKit, and how much naturally belongs in the app delegate and view controller?

On iOS, there will definitely be many bits and pieces which need to go in the app delegate in relation to implementing this stuff from UIApplicationDelegate:

– applicationDidBecomeActive:
– applicationWillResignActive:
– applicationDidEnterBackground:
– applicationWillEnterForeground:
– applicationWillTerminate:
– applicationDidFinishLaunching:
ghost commented 10 years ago

Today I added offline map downloading controls and a progress indicator to the iOS sample app. It looks like: mbx sample app offline map gui

The one on the left is the offline map downloader screen, and the one on the right shows how the progress bar will stay up until download is done or cancelled.

Note: This isn't actually downloading maps yet! My goal was to work on the GUI and the interface between the iOS view controller and [MBXOfflineMapDownloader sharedOfflineMapDownloader] (i.e. begin/cancel/resume/suspend and the delegate callbacks). MBXOfflineMapDownloader is fake in the sense that it's just using a timer to drive the progress indicator, but the view controller stuff is all real. By that I mean the UI behavior for begin, cancel, suspend/resume, and the progress indicator all work.

ghost commented 10 years ago

Since yesterday, I've started getting an EXEC_BAD_ACCESS once in a while when I start the app. The crashing seems to have pretty similar symptoms to what @lightandshadow68 reported at https://github.com/mapbox/mbxmapkit/issues/26

Here's a stack trace

#0  0x00000001960601dc in objc_msgSend ()
#1  0x0000000192f562cc in __50-[VKRasterOverlayTileSource invalidateRect:level:]_block_invoke_2 ()
#2  0x0000000192e07dd8 in -[VKTileKeyMap enumerateKeysAndObjectsUsingBlock:] ()
#3  0x0000000192e07d64 in -[VKTilePool enumerateKeysAndObjectsUsingBlock:] ()
#4  0x0000000192f56224 in __50-[VKRasterOverlayTileSource invalidateRect:level:]_block_invoke ()
#5  0x0000000196628014 in _dispatch_call_block_and_release ()
#6  0x0000000196627fd4 in _dispatch_client_callout ()
#7  0x000000019662b1dc in _dispatch_main_queue_callback_4CF ()
#8  0x0000000189b0a62c in __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ ()
#9  0x0000000189b0896c in __CFRunLoopRun ()
#10 0x0000000189a496d0 in CFRunLoopRunSpecific ()
#11 0x000000018f72dc0c in GSEventRunModal ()
#12 0x000000018cb7afdc in UIApplicationMain ()
#13 0x000000010009eb38 in main at [...]/mbxmapkit/Sample Project/MBXMapKit iOS Demo v030/main.m:16
#14 0x0000000196643aa0 in start ()

Seems like there's a good chance I messed something up in the last few commits and triggered an MKMapKit bug. A quick search for VKRasterOverlayTileSource invalidateRect:level: EXEC_BAD_ACCESS yields a couple pages worth of results. It doesn't seem like this is an isolated thing.

ghost commented 10 years ago

The crashing problem seems to have been caused by threading issues with the delegate callbacks, but I think I've fixed that now. See https://github.com/mapbox/mbxmapkit/issues/77#issuecomment-38336533.

lightandshadow68 commented 10 years ago

I wanted to add my current project as a use case for offline map caching with MBXMapView.

To support true offline support, I've created an adaptor for RMTileCache so it can be used with MBXMapView. When the user logs in, the app calls beginBackgroundCacheForTileSource:southWest:northEast:minZoom:maxZoom: for each geographical region associated with the user so they can view maps of these regions even if the device goes offline. Tiles for all regions are stored in the same SQL file created by RMTileCache.

Caching tiles for each location occurs in a serial. When a tile cache operation completes, the app checks if any uncached locations remain, asks RMTileCache to cache those tiles as well, etc. This occurs in a headless fashion. When MBXMapViews are eventually displayed in the app, they are associated with this my adaptor class and the file is persisted to disk between launches of the app.

Currently, we do not have logic in place to limit users from panning and zooming the map outside the range of cached tiles. If this occurs and the device is online, missing tiles are downloaded over the network and stored in the default cache associated with the MBXMapView, rather than the file associated with the RMTileCache. If the device is offline, exposed areas are blank.

lightandshadow68 commented 10 years ago

@wsnook The radar I filed with Apple for that issue was marked as a duplicate, which is still open as of this comment.

13556600 User interface anomaly in OS X

State: Duplicate Product: OS X Rank: 3 - Medium Duplicate of 10535951 (Open)

ghost commented 10 years ago

@lightandshadow68 Thanks for the info on your radar and the offline map use case.

ghost commented 10 years ago

I just committed an update to the OS X demo app (offline download controls & progress bar) which brings it back up to the point of matching the iOS demo app. Now I'm going to start focusing my attention just on the iOS side for a while and see what happens over at #78 regarding whether we want to keep supporting OS X going forward.

ghost commented 10 years ago

Summing up where things stand on this epic ticket...

  1. This ticket is a continuation of work that started in #61 and #71.
  2. The plan was to figure out, in code, what a new MBXMapKit api could/should look like in order to better solve problems related to caching and asynchronous loading of map metadata.
  3. Online performance caching using NSURLCache is done.
  4. Asynchronous loading of metadata, and an iOS sample app to demonstrate several ways of using it, are done.
  5. A mechanism to download and persistently store offline map resources (tiles, json, and markers) is in progress.

For the offline map downloader, the main obvious next steps are:

ghost commented 10 years ago

I'm changing direction a bit with the offline map downloading in response to some chat and skype conversations with @incanus today...

A couple weeks ago, about half way up this thread, we talked about how background downloading ought to work, see https://github.com/mapbox/mbxmapkit/issues/73#issuecomment-38091960. I came away with the impression that @incanus felt like we ought to be using NSURLDownloadTask in a background NSURLSession (from Apple's docs: "When you use download tasks in background sessions, these downloads continue even when your app is suspended or is otherwise not running."), and I've been working toward implementing that.

After today's conversations, it seems that it will actually be sufficient just to have a mechanism which does offline map downloads in an operation queue while the app is running. Neither of us is aware of any compelling reason to wade into the complexity of using background sessions with NSURLSession NSURLDownloadTask.

So, for the purposes of downloading tiles, json, and markers to use as an offline map, I'm re-defining "background downloading" to mean:

  1. It should be possible for an app to have an offline map download going in the background while allowing interaction with some other map on screen.
  2. The background downloading mechanism should persist its state so that when the app wakes up or re-launches after becoming inactive (somebody presses the home button, iOS kills the app due to memory pressure, etc), it will be able to resume the offline map download from the point it left off.
ghost commented 10 years ago

Status update summarizing my recent commits above:

  1. The MBXOfflineMapDatabase class is theoretically complete and ready, but it needs further testing on the sqlite side. It's usable for initializing an MBXRasterTileOverlay layer, and the sample app is set up to do that.
  2. The plumbing in the sample app to restore saved state from disk when the app is re-launched is theoretically complete. That includes resuming a suspended download job and populating the list of completed offline maps. The resume-a-download-on-re-launch feature has been tested against the timer based fake progress indicator that's currently still built into MBXOfflineMapDownloader, so it ought to work once the real stuff is ready.
  3. When MBXOfflineMapDownloader receives a begin message, it's currently set up to create a real metadata dictionary and a fake array of tiles to be downloaded (fake = from an array literal instead of a calculation). As of last night, it can now write the metadata and list of tiles out to an sqlite database. The database schema is:
CREATE TABLE metadata (name text, value text);
CREATE UNIQUE INDEX name ON metadata (name);
CREATE TABLE resources (url text, status text, data blob);
CREATE UNIQUE INDEX url ON resources (url);

The status column of the resources table is how MBXOfflineMapDownloader will keep track of which urls have been processed and which ones still need to be requested. Building the mechanism to do that is next up.

ghost commented 10 years ago

The next step is to build a pipeline for reading some urls from the db, putting those urls in the download queue, writing the downloaded files back to the db, and keeping the process rolling with additional batches of urls until everything has been downloaded.

Building that pipeline is going to take some thought about how to balance the following constraints and questions in a reasonable way, according to the "good enough" principle:

  1. How many urls should be read and processed in one batch? One is definitely too few, and 100 is quite possibly too many. In any case, the system should be robust enough to process maps with many thousands of tiles.
  2. If we've got the device's radio turned on, we should be making the most of the battery usage by running several downloads in parallel. That means we need to keep at least a small backlog in the NSURLSession's download queue.
  3. It needs to be possible to rapidly and cleanly suspend the processing and to resume it again later.
incanus commented 10 years ago

You may also find use in exploring hosts [a-d].tiles.mapbox.com and -[NSURLSessionConfiguration HTTPMaximumConnectionsPerHost].

ghost commented 10 years ago

With the maximum connections per host thing, I think that just using NSURLSession will work fine and there aren't really any problematic questions at that level (i.e. how do we download 4/8/16 files at once). It gets more complicated to make sure that NSURLSession consistently has a reasonable number of urls waiting in its queue to be downloaded.

I'm expecting that the NSURLSession will be managing a download queue with a typical backlog of perhaps tens of items waiting to be downloaded in parallel. On the other hand, there might be a backlog in the sqlite offline map database of 1,000 (or 10,000!) items waiting to be downloaded.

Throwing 10,000 urls at an NSURLSession seems like a pretty good way to cause memory problems, and on the other end, letting NSURLSession repeatedly drain the backlog down to nothing will lead to lots of time when less than the full number of possible simultaneous downloads are going at once (i.e. takes longer and uses more power).

I'm thinking the solution will look like two queues, the small one (NSURLSession) will be for the purpose of keeping simultaneous downloads going while the radio is turned on. The larger one (offline map sqlite db plus some logic) will be to make sure NSURLSession has a backlog that's not too large and not too small, for the purpose keeping memory usage and disk activity within reasonable limits.

ghost commented 10 years ago

Note to self: To get some order of magnitude numbers for how many tiles people might try downloading, try calculating the tiles for these regions, zoom 0-17: