Cached tiles vanish when scene is reloaded

westnordost commented 4 years ago

Environment

OS: Android (any version) Tangram-Version: 0.12

Reproduction

View the map
Turn airplane mode on. New tiles won't be loaded, but everything that is in cache is shown
Call MapController::loadSceneFileAsync

Result: The map is completely gone. Expected Result: The cached tiles are still shown with the new scene.

Other

Actually for my use case I do not need to reload the whole scene but just do a scene update. Since the (shortcut) function was removed since 0.10, I call loadSceneFileAsync but I read that this amounts to the same as a scene reload before anyway.

However, this problem did not exist in 0.9.6.

Related issue in StreetComplete: https://github.com/westnordost/StreetComplete/issues/1719

westnordost commented 4 years ago

This happens even with a cache control set to

CacheControl.Builder()
        .maxStale(365, TimeUnit.DAYS)
        .minFresh(365, TimeUnit.DAYS)
        .build()

on the client side. In case this is related to #1777

matteblair commented 4 years ago

Hey @westnordost - the behavior you're describing is definitely not intended. I looked into this and modified one of the sample apps to try and reproduce the error, but when I try it the cache behaves as expected. I used the "sceneupdates" sample on this branch: https://github.com/tangrams/tangram-android-demos/tree/test-tilecache-sceneupdate

Here's what I did:

Start the app with network on
Zoom in from the world map to a city until buildings are visible
Turn the phone's network access off
Tap the "buildings" checkbox to trigger a scene update
Pan around to confirm that new tiles in un-visited areas don't load
Zoom back out to confirm that tiles in visited areas are still cached

There may be something else going on in your case that is preventing the cache from working as intended. So far I am not able to reproduce this behavior.

westnordost commented 4 years ago

Ok I'll research what triggers it exactly asap!

westnordost commented 4 years ago

For me, it shows only a grey screen. So I switched out the nextzen tileserver in line 439 of bubble-wrap-style.yaml with https://tiles.map-data.de/vector/v1/512/all/{z}/{x}/{y}.mvt. Then it showed a map.

I can confirm that it is not reproducible with the code in https://github.com/tangrams/tangram-android-demos/tree/test-tilecache-sceneupdate

If I comment out updateList.add(apiKeySceneUpdate); in line 84, it becomes reproducible.

map-data.de at the moment doesn't need an api key and it loads fine without it when a network connection is available.

matteblair commented 4 years ago

Aha, the API key is a very good find! I now believe that the behavior you're seeing is a result of a missing API key after the scene change.

The interface change from updateSceneAsync to loadSceneAsync has an important difference in behavior: with updateScene any previous updates would still be applied, but with loadScene the scene is loaded in a pristine state with no updates applied[1]. The sceneupdates sample app demonstrates how you would apply repeated scene changes with a cumulative list of updates.

In short, whenever you call loadSceneAsync you should pass in the list of all the updates that your scene needs.

[1] = There are arguments for the merits of both of these models, but the big downside to updateScene is that it creates an implicit state for the scene, making it impossible for a user to programmatically determine the scene's current state (i.e. what updates have been applied). The only way to be certain about the value of all the updates is to re-apply updates for the set of all values that could have been changed. loadScene makes this clear and encourages the user to maintain a list of scene updates that explicitly mirrors the values changed in the current scene.

westnordost commented 4 years ago

Why is the behavior the result of a missing API key? As said, for map-data.de, I don't use and don't set any api key in the first place.

matteblair commented 4 years ago

Maybe I misunderstood your earlier comment. What I understood was that when you switched your source URL to map-data.de (which does not use an API key) the tile cache worked as expected after a scene change. Since the nextzen.org URL does use an API key and it causes the tile cache to not work as expected after a scene change, the problem seems to lie with the use of an API key with the source URL.

Now, since the situation we're trying to fix is an offline state, it may not be obvious why the API key matters. For most tile servers, including nextzen.org, the API key is applied as a URL parameter. That means if the API key in a scene changes (or becomes empty) then the URL used to request a tile will also change, and this new URL will not have a cache entry. In short, changing the API key (or any URL parameter) for a tile source will invalidate its entire cache.

westnordost commented 4 years ago

Ah, so this was the misunderstanding.

I meant to say that the tangram demo did not show a map at all for me, probably because I needed to set a nextzen api key in the build config. Instead of doing that, I just exchanged the tileserver url to get it to run. With this new tileserver url, I could confirm that the reported issue is not reproducible in that app.

I played around a bit and found out that if I don't set the api key update at all (neither on start nor on a subsequent update), the issue becomes reproducible.

Now, I tried it again and it is not 100% reproducible with this, so I tried some more. What I do now is:

Zoom to an area where buildings are displayed
Switch on airplane mode
toggle the buildings checkbox. Everything works as expected
now pan the offline map to near the border of what is still loaded on a slightly lower zoom level
zoom in again at that point: everything displays correctly
again toggle the buildings checkbox
everything is gone, doesn't come back
but if you zoom out and then in again, the tiles get displayed again

Not sure anymore if this is the same issue I am experiencing with my app. But anyway it is related to it.

Suspicion: The tile near the border of what is still loaded is only available in lower zoom levels. When the scene is reloaded, tangram only tries to get the tile in the current zoom level and won't fall back to a lower one automatically. Only after zooming out and then zooming in again, it is displayed.

peternewman commented 4 years ago

FWIW I can confirm zooming in and out in SC fixes some of my offline issues/makes bits of map appear when there is none when offline. Presumably this is a combination of SC slightly modifying the zoom level of the map when completing a quest and as @westnordost says Tangram not using a lower zoom level cached tile by default.

westnordost commented 4 years ago

Ok turns out that the caching did not work for my app, see #2145. So what remains of this report is the repro from my prior comment.

matteblair commented 4 years ago

Oh I definitely misunderstood the situation haha - thanks for clarifying

The issue with inheriting from DefaultHttpHandler in Kotlin seems like the bigger problem for StreetComplete, though your testing shows that there are also other edge cases where the tile resolution logic should make better use of the cache. I'll work on addressing the inheritance issue first.

westnordost commented 2 years ago

I'm reading this issue again after 2 years and I don't understand what the remaining issue was. Apparently the issue initially reported turned out to be my own fault. The other issue I don't understand and it is buried in a long interchange. So, I'll close this issue and if (something like that) turns up again, will open a new issue for that.

tangrams / tangram-es