Add disk caching for tiles

[david-rhodes]

Similar to caching tiles in memory, we should also support caching of tiles to disk.

I imagine this would be a different implementation of IAsyncRequest.

In solving for this, we have to keep in mind the Mapbox ToS:

You may cache Map Assets on end-user devices (e.g., laptops, smartphones, or tablets) for offline use, but each device must populate its cache using direct requests to the Mapping APIs and content from a cache may only be consumed by a single end user. On mobile devices, you may only cache up to the limits set in the Mobile SDKs, and you may not circumvent or change those limits. You may not redistribute Map Assets, including from a cache, by proxying, or by using a screenshot or other static image instead of accessing Map Assets through the Mapping APIs. You may not scrape or download Map Assets in bulk for any purpose other than offline caching on a single end user’s device.

An app can download multiple regions for offline use, but the total offline download is capped at a maximum tile count “ceiling” across all downloaded regions. The tile ceiling is set to 6,000 tiles by default but can be raised for paid plans.

[isiyu]

@david-rhodes this is "phase 3" outlined here: https://github.com/mapbox/mapbox-sdk-cs/issues/66#issue-222569980

"file system" and "on disk" are effectively the same. Based on the TOS, we can use a system similar to Mapbox mobile's native SDKs, which is pre-downloaded areas approx. the size of London

(The tile ceiling is set to 6,000 tiles by default but can be raised for paid plans.)

[david-rhodes]

@isiyu Ticketed here because I suspect it will be a unity layer implementation (accessing files from various OSs).

[derekdominoes]

Here is my implementation of local file based caching. I wanted to validate and see if it can be efficiently done. It is in no way optimal. It is fully implemented in MapController.cs and so architecturally not likely the correct way in that it might be better to subclass or add an interface to MapController.

I use it with the SlippyVectorTerrain sample.

To get it to work you must first repair a bug in MapImageFactory.cs:83. Assignement of tile.ImageData causes ImageDataChanged event to be fired before ImageData is loaded Fix this in your version of MapImageFactory.cs

VectorData has not been set in the current UnityTile so vectors are not cached.

Try it out and me know what you think.

`namespace Mapbox.Unity.MeshGeneration { using UnityEngine; using System.Collections.Generic; using Mapbox.Unity.MeshGeneration.Data; using Mapbox.Unity; using Mapbox.Platform; using Mapbox.Unity.Utilities; using Utils; using System; using System.IO;

/// <summary>
/// MapController is just an helper class imitating the game/app logic controlling the map. It creates and passes the tiles requests to MapVisualization.
/// </summary>
public class MapController : MonoBehaviour, IFileSource, IAsyncRequest
{
    public static RectD ReferenceTileRect { get; set; }
    public static float WorldScaleFactor { get; set; }

    public MapVisualization mapVisualization;
    public float TileSize = 100;

    [SerializeField]
    private bool _snapYToZero = true;

    [Geocode]
    public string LatLng;
    public int Zoom;
    public Vector4 Range;

    private MapVisualization localMapVisualization;
    private string cacheRoot;
    private GameObject _root;
    private Dictionary<Vector2, UnityTile> _tiles;

    /// <summary>
    /// Resets the map controller and initializes the map visualization
    /// </summary>
    public void Awake() {
        cacheRoot = (Application.isEditor ? Application.dataPath.Substring( 0, Application.dataPath.Length - 7 ) : Application.persistentDataPath) + @"/cache/v1/";
        // Need unique Factorie instances in our caches MapVisualization so that they can have a IFileSource not shared with the MapboxAccess MapVisualization
        // Clone the same factories that have been specified in the Unity inspector for MapVisualization
        localMapVisualization = ScriptableObject.CreateInstance<MapVisualization>();
        localMapVisualization.Factories = new List<Factories.Factory>();
        foreach (var factory in mapVisualization.Factories) {
            var clone = ScriptableObject.Instantiate( factory );
            localMapVisualization.Factories.Add( clone );
        }
        localMapVisualization.Initialize( this );   // Our caches IFileSoure will be handled by this classes implementation MapControler.IFileSource.Request
        mapVisualization.Initialize(MapboxAccess.Instance);
       _tiles = new Dictionary<Vector2, UnityTile>();
    }

    public void Start()
    {
        Execute();
    }

    /// <summary>
    /// Pulls the root world object to origin for ease of use/view
    /// </summary>
    public void Update()
    {
        if (_snapYToZero)
        {
            var ray = new Ray(new Vector3(0, 1000, 0), Vector3.down);
            RaycastHit rayhit;
            if (Physics.Raycast(ray, out rayhit))
            {
                _root.transform.position = new Vector3(0, -rayhit.point.y, 0);
                _snapYToZero = false;
            }
        }
    }

    public void Execute()
    {
        var parm = LatLng.Split(',');
        Execute(double.Parse(parm[0]), double.Parse(parm[1]), Zoom, Range);
    }

    public void Execute(double lat, double lng, int zoom, Vector2 frame)
    {
        Execute(lat, lng, zoom, new Vector4(frame.x, frame.y, frame.x, frame.y));
    }

    public void Execute(double lat, double lng, int zoom, int range)
    {
        Execute(lat, lng, zoom, new Vector4(range, range, range, range));
    }

    /// <summary>
    /// World creation call used in the demos. Destroys and existing worlds and recreates another one. 
    /// </summary>
    /// <param name="lat">Latitude of the requested point</param>
    /// <param name="lng">Longitude of the requested point</param>
    /// <param name="zoom">Zoom/Detail level of the world</param>
    /// <param name="frame">Tiles to load around central tile in each direction; west-north-east-south</param>
    public void Execute(double lat, double lng, int zoom, Vector4 frame)
    {
        //frame goes left-top-right-bottom here
        if (_root != null)
        {
            foreach (Transform t in _root.transform)
            {
                Destroy(t.gameObject);
            }
        }

        _root = new GameObject("worldRoot");

        var v2 = Conversions.GeoToWorldPosition(lat, lng, new Vector2d(0, 0));
        var tms = Conversions.MetersToTile(v2, zoom);
        ReferenceTileRect = Conversions.TileBounds(tms, zoom);
        WorldScaleFactor = (float)(TileSize / ReferenceTileRect.Size.x);
        _root.transform.localScale = Vector3.one * WorldScaleFactor;

        for (int i = (int)(tms.x - frame.x); i <= (tms.x + frame.z); i++)
        {
            for (int j = (int)(tms.y - frame.y); j <= (tms.y + frame.w); j++) {
                var tile = new GameObject( "Tile - " + i + " | " + j ).AddComponent<UnityTile>();
                _tiles.Add( new Vector2( i, j ), tile );
                tile.Zoom = zoom;
                tile.RelativeScale = Conversions.GetTileScaleInMeters( 0, Zoom ) / Conversions.GetTileScaleInMeters( (float)lat, Zoom );
                tile.TileCoordinate = new Vector2( i, j );
                tile.Rect = Conversions.TileBounds( tile.TileCoordinate, zoom );
                tile.transform.position = new Vector3( (float)(tile.Rect.Center.x - ReferenceTileRect.Center.x), 0, (float)(tile.Rect.Center.y - ReferenceTileRect.Center.y) );
                tile.transform.SetParent( _root.transform, false );
                if (InCache( tile ))
                    localMapVisualization.ShowTile( tile );
                else {
                    tile.ImageDataChanged += Tile_ImageDataChanged;
                    tile.HeightDataChanged += Tile_HeightDataChanged;
                    mapVisualization.ShowTile( tile );
                }
            }
        }
    }

    /// <summary>
    /// Used for loading new tiles on the existing world. Unlike Execute function, doesn't destroy the existing ones.
    /// </summary>
    /// <param name="pos">Tile coordinates of the requested tile</param>
    /// <param name="zoom">Zoom/Detail level of the requested tile</param>
    public void Request(Vector2 pos, int zoom)
    {
        if (!_tiles.ContainsKey(pos))
        {
            var tile = new GameObject("Tile - " + pos.x + " | " + pos.y).AddComponent<UnityTile>();
            _tiles.Add(pos, tile);
            tile.transform.SetParent(_root.transform, false);
            tile.Zoom = zoom;
            tile.TileCoordinate = new Vector2(pos.x, pos.y);
            tile.Rect = Conversions.TileBounds(tile.TileCoordinate, zoom);
            tile.RelativeScale = Conversions.GetTileScaleInMeters(0, Zoom) / Conversions.GetTileScaleInMeters((float)Conversions.MetersToLatLon(tile.Rect.Center).x, Zoom);
            tile.transform.localPosition = new Vector3((float)(tile.Rect.Center.x - ReferenceTileRect.Center.x),
                                                       0,
                                                       (float)(tile.Rect.Center.y - ReferenceTileRect.Center.y));
            if (InCache( tile ))
                localMapVisualization.ShowTile( tile );
            else {
                tile.ImageDataChanged += Tile_ImageDataChanged;
                tile.HeightDataChanged += Tile_HeightDataChanged;
                tile.VectorDataChanged += Tile_VectorDataChanged;
                mapVisualization.ShowTile( tile );
            }
        }
        else {
            var tile = _tiles[ pos ];
            if (tile.ImageDataState == Enums.TilePropertyState.Error)
                Debug.Log( "Offline Tile" );    // Tile previously loaded when offline and data for it was not cached and couldn't be retieved from the internet
        }
    }

    // Cache Support ---------------------------------------

    private void Tile_HeightDataChanged( UnityTile sender, object param ) {
        CacheImage( sender, true );
    }

    private void Tile_ImageDataChanged( UnityTile sender, object param ) {
        // Bug: MapImageFactory.cs:83 assignement of tile.ImageData causes ImageDataChanged event to be fired before ImageData is loaded; fix this in your version of MapImageFactory.cs
        CacheImage( sender );
    }

    private void Tile_VectorDataChanged( UnityTile sender, object param ) {
        CacheVectors( sender ); // Never gets called in the current implementation of MeshFactory.CreateMeshes
                                // VectorTile.data is uncompressed in VectorTile.ParseTileData. No vector data seems to be stored in the UnityTile.VectorData
                                // To make this work, the compressed data needs to be stored in UnityTile.VectorData
    }

    // Implements IFileSource
    IAsyncRequest IFileSource.Request( string uri, Action<Response> callback ) {
        // uir format = https://api.mapbox.com/styles/v1/mapbox/satellite-v9/tiles/14/3402/6200
        // or           https://api.mapbox.com/v4/mapbox.terrain-rgb/14/3402/6200.pngraw
        // or           https://api.mapbox.com/v4/mapbox.mapbox-streets-v7/14/3402/6200.vector.pbf
        int lonStart = uri.LastIndexOf( '/' );
        int latStart = uri.Substring( 0, lonStart - 1 ).LastIndexOf( '/' );
        int scaleStart = uri.Substring( 0, latStart - 1 ).LastIndexOf( '/' );
        string localPath = cacheRoot + uri.Substring( scaleStart );
        if (uri.EndsWith( ".pngraw" ))
            localPath = localPath.Substring( 0, localPath.Length - 7 ) + @"/H.png";
        else if (uri.EndsWith( ".vector.pbf" )) {
            localPath = localPath.Substring( 0, localPath.Length - 11 ) + @"/V.pbf";
            //Debug.Log( "Vector cache unhandled: " + uri );
            return this;
        }
        else
            localPath += @"/I.jpg";

        using (var fs = File.Open( localPath, FileMode.Open )) {
            int fileLen = (int)fs.Length;
            byte[] image = new byte[ fileLen ];
            fs.BeginRead( image, 0, fileLen, ar => {
                fs.EndRead( ar );
                callback.Invoke( new Response() { Data = image } );
            }, callback );
        }
        return this;
    }

    void CacheImage( UnityTile tile, bool height = false ) {
        try {
            var localPath = CachedTilePath( tile );
            CreatePath( localPath );
            localPath += height ? @"/H.png" : @"/I.jpg";

            using (var fs = File.Open( localPath, FileMode.Create )) {
                var image = height ? tile.HeightData.EncodeToPNG() : tile.ImageData.EncodeToJPG();
                fs.BeginWrite( image, 0, image.Length, ar => {
                    fs.EndWrite( ar );
                },  null );
            }
        }
        catch (Exception e) {
            Debug.Log( e.Message );
        }
    }

    void CacheVectors( UnityTile tile ) {
        try {
            var localPath = CachedTilePath( tile );
            CreatePath( localPath );
            localPath += @"/V.pbf";

            using (var fs = File.Open( localPath, FileMode.Create )) {
                string vectorsString = tile.VectorData;
                var vectors = System.Text.Encoding.ASCII.GetBytes( vectorsString );
                fs.BeginWrite( vectors, 0, vectors.Length, ar => {
                    fs.EndWrite( ar );
                }, null );
            }
        }
        catch (Exception e) {
            Debug.Log( e.Message );
        }
    }

    bool InCache( UnityTile tile ) {
        return Directory.Exists( CachedTilePath( tile ) );
    }

    string CachedTilePath( UnityTile tile ) {
        var coords = tile.TileCoordinate;
        return cacheRoot + tile.Zoom.ToString() + @"/" + coords.x + @"/" + coords.y;
    }

    void CreatePath( string path ) {
        var parts = path.Split( new char[] { '/' } );
        string startPath = "";
        foreach (var part in parts ) {
            startPath += part;
            if (!Directory.Exists( startPath ))
                Directory.CreateDirectory( startPath );
            startPath += "/";
        }
    }

    // Implements IAsyncRequest
    void IAsyncRequest.Cancel() {
    }

}

}`

[david-rhodes]

@derekdominoes Cool! We've recently added memory caching, and it's in a layer below MapController. This allows us to cache data regardless where the query comes from. Check MapboxAccess.cs in develop to see how it is implemented. We don't yet have disk caching, but the abstractions should easily enable it (and we'll be working on this soon).

One thing to consider with disk caching is how often you access the disk. You don't want to hit too frequently or your frame rate will suffer and you will drain more battery (on mobile).

[david-rhodes]

Initial implementation of disk caching has been added! https://github.com/mapbox/mapbox-unity-sdk/pull/125