LocalTileFile implementation

[360CAS]

Target Use case

A localTileFile layer would make it far easier for general users to work with deck.gl, by simplifying the process of accessing local {z}/{x}/{y} tile data. Being able to create application that work from a single machine particularly with sensitive local data is a major use case

Proposed feature

Suggest creating a LocalTileFile layer similar to what has been done recently in Ipyleaflet see https://ipyleaflet.readthedocs.io/en/latest/api_reference/local_tile_layer.html

To Do List

• [ ] Add label and assign to milestone
• [ ] Coding
• [ ] Doc update
• [ ] What’s new update
• [ ] Test

[kylebarron]

I'm not clear on what you're suggesting. Are you referring to the JavaScript deck.gl or the Python pydeck? The browser has limitations on being able to load files from the user's file system. The easiest way in general to work with local files is to start a local file server.

Go to your directory of tile data, run python -m http.server (for Python 3), then you'll be able to access your data via http://localhost:8080/*.

[360CAS]

Hi Kyle,

Was suggesting for deck.gl and by extension pydeck. The CoRs restrictions on browsers do indeed make doing things difficult for people that need to work locally.

Given the power of deck.gl I know many large organisations that would like to be able to use it for data science and in general these large corps who are the kind of folk that have volumes of data from which they wish to get valuable insight.

However in my experience of large corps, getting permission to run your own http server from IT would be be very painful. So while one may be able to get a stack like deck.gl though audit eventually, you would stumble at the user being able to deploy it. The very idea of mixing data access with webstack access give IT the Heebie-jeebies.

It is possible to hack around this to make it work, but it is a pain. We have managed to do this with Open layers and independently with Ipyleaflet though its "localtilefile layer" but it has taken a lot of digging to find the solutions.

Making the process easier for folks looking to use deck.gl with local data, with reduced or zero access to external connection would in my opinion drive adoption.

[kylebarron]

ipyleaflet does this by using Python to load your files and then passing that data (via localhost, I should add, since Jupyter is running on localhost) to JS. The browser cannot access data on a user's file system without the user specifically selecting it in a dialog box. This is also separate from CORS restrictions. (Otherwise you'd have websites loading and exfiltrating all users' data all the time!)

Because the file system is sandboxed, a web app cannot access another app's files. You also cannot read or write files to an arbitrary folder (for example, My Pictures and My Documents) on the user's hard drive.

https://developer.mozilla.org/en-US/docs/Web/API/File_and_Directory_Entries_API/Introduction#sandbox

In JavaScript, there is no way around using some sort of server to pass data to the browser. You could potentially use Python for this, since Python can access your entire filesystem, but there are other impediments to implementing the TileLayer in Pydeck, namely that there's no way to define a JS function for getTileData in Python.

[360CAS]

Thanks Kyle,

So disclaimer I know nothing about js or react.

But via randomly pushing key I did get it to work in JS after putting the data in the in the project folder which is a horrible hack but effective.

This is what worked for me based on one of the examples

import React from 'react';
import {render} from 'react-dom';
import {load} from '@loaders.gl/core';
import DeckGL from '@deck.gl/react';
import {MapView} from '@deck.gl/core';
import {TileLayer} from '@deck.gl/geo-layers';
import {BitmapLayer, PathLayer} from '@deck.gl/layers';

// Viewport settings
const INITIAL_VIEW_STATE = {
//  target: [500, 500, 0],
//  rotationX: 0,
//  rotationOrbit: 0,
//  zoom: 1
    latitude: 7,
    longitude: 7,
    zoom: 2,
    maxZoom: 5,
    maxPitch: 20,
    bearing: 0
};

/* global window */
const devicePixelRatio = (typeof window !== 'undefined' && window.devicePixelRatio) || 1;

function getTooltip({tile}) {
  return tile && `tile: x: ${tile.x}, y: ${tile.y}, z: ${tile.z}`;
}

export default function App({showBorder = false, onTilesLoad = null}) {
  const tileLayer = new TileLayer({
    // https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames#Tile_servers
    data: [  './tiles/{z}/{x}/{y}.png' ],

    maxRequests: 20,

    pickable: true,
    onViewportLoad: onTilesLoad,
    autoHighlight: showBorder,
    highlightColor: [60, 60, 60, 40],
    minZoom: 2,
    maxZoom: 5,
    _getTileData: tile => {
      if (tileWithinBounds(tile.x, tile.y, tile.z)) { return load(tile.url); }
      return null;
    },
    tileSize: 256 / devicePixelRatio,

    renderSubLayers: props => {
      const {
        bbox: {west, south, east, north}
      } = props.tile;

      return [
        new BitmapLayer(props, {
          data: null,
          image: props.data,
          bounds: [west, south, east, north]
        }),
        showBorder &&
          new PathLayer({
            id: `${props.id}-border`,
            visible: props.visible,
            data: [[[west, north], [west, south], [east, south], [east, north], [west, north]]],
            getPath: d => d,
            getColor: [255, 0, 0],
            widthMinPixels: 4
          })
      ];
    }
  });

  return (
    <DeckGL
      layers={[tileLayer]}
      views={new MapView({repeat: false})}
      initialViewState={INITIAL_VIEW_STATE}
      controller={true}
      getTooltip={getTooltip}
    />
  );
}

export function renderToDOM(container) {
  render(<App />, container);
}`

[kylebarron]

putting the data in the in the project folder

I think that's the only way it could conceivably work in JS, and I'm not sure whether that's the JS bundling engine doing the work for you. If you replace the local path with an absolute path on your hard drive, I doubt it'll work.

[Pessimistress]

As @kylebarron has pointed out, JavaScript is not allowed to access local files from the browser. There is no layer implementation, or anything in the deck.gl JS library that can work around it. Copying the tiles into the project folder works because Webpack dev server, or whatever dev setup you are using, serves all of the files in the working directory as static assets.

However, Python can access the local file system. If you only need this to work with pydeck, then pydeck should be able to spin up a local server to serve these files.

[360CAS]

Thanks @kylebarron @Pessimistress

Python seems like the way to go but there are other challenges in pydeck around getting tilefiles to work.

Not sure if it makes a difference but with tile files we are also not accessing a single file but traversing a tree in a directory, unless we could load the whole tilefile as maybe an awkward array?

How about passing the Tilefile to the script during run time as user input, same way you would upload a file? This would be like chose my map?

[Pessimistress]

@360CAS Can you clarify what exactly is the user experience you are proposing?

• Use JavaScript and tiles both hosted on a static HTTP server - how works today
• Use pydeck with tiles served from a local directory - possible, with changes to pydeck
• Use JavaScript, drag and drop a local directory into the browser - possible, but it needs to be implemented in your own application

[Assistedevolution]

@Pessimistress I think I was looking to simplify the process of achieving points 2&3 for people using local data. i.e. .

• Use pydeck with tiles served from a local directory - possible, with changes to pydeck

• Use JavaScript, drag and drop a local directory into the browser - possible, but it needs to be implemented in your own application.

So either though documented examples or code changes. Support for tiles in pydeck would be a great first step

Thanks for listening

[geniuskim05]

Hi @360CAS @Pessimistress! I have a simple question. Is tileWithinBound function?? or Do I make function? getTileData: tile => { if (tileWithinBounds(tile.x, tile.y, tile.z)) { return load(tile.url); } return null; },

[kylebarron]

@geniuskim05 that's unrelated to this issue, so I'd suggest making a discussion question instead for the future. The TileLayer gives a bbox prop to getTileData, which you can intersect with your bounds to determine whether to load the tile

[hokieg3n1us]

After #6121 is merged, you'll be able to use local XYZ tilesets (though, the same approach could be used to utilize WMS tilesets from a local GeoServer) pretty simply using Pydeck.

You'll be able to pass the Deck class the the local tiles as a custom_map_style (a dictionary that models the Mapbox style specification) like below:

custom_map_style = {
    "version": 8,
    "sources": {
        "local": {
            "type": "raster",
            "tiles": [
                "http://localhost:8080/tiles/{z}/{x}/{y}.png"
            ],
            "scheme": "xyz",
            "tileSize": 256
        },
    },
    "layers": [
        {
            "id": "local",
            "type": "raster",
            "source": "local",
            "source-layer": "local",
            "minzoom": 0,
            "maxzoom": 18
        }
    ]
}

r = pdk.Deck(layers=[layer], map_style=custom_map_style, map_provider="mapbox")

Then you can serve your local tiles by running the Python HTTPServer from the parent directory. Below is a simple example that allows CORS requests. This is useful if you export your Deck to a local HTML file like I usually do.

python server.py 127.0.0.1 8080

server.py

from http.server import HTTPServer, SimpleHTTPRequestHandler
import sys

class CORSRequestHandler(SimpleHTTPRequestHandler):

    def end_headers(self):
        self.send_header('Access-Control-Allow-Origin', '*')
        self.send_header('Access-Control-Allow-Methods', '*')
        self.send_header('Access-Control-Allow-Headers', '*')
        self.send_header('Cache-Control', 'no-store, no-cache, must-revalidate')
        return super(CORSRequestHandler, self).end_headers()

    def do_OPTIONS(self):
        self.send_response(200)
        self.end_headers()

host = sys.argv[1] if len(sys.argv) > 2 else '0.0.0.0'
port = int(sys.argv[len(sys.argv) - 1]) if len(sys.argv) > 1 else 8080

print("Listening on {}:{}".format(host, port))
httpd = HTTPServer((host, port), CORSRequestHandler)
httpd.serve_forever()

This image above is a demonstration, actually overlaying a local tile set that I created using datashader, overlaid on tiles provided by OpenStreetMap.

[jslorrma]

After #6121 is merged, you'll be able to use local XYZ tilesets (though, the same approach could be used to utilize WMS tilesets from a local GeoServer) pretty simply using Pydeck.

You'll be able to pass the Deck class the the local tiles as a custom_map_style (a dictionary that models the Mapbox style specification) like below:

custom_map_style = {
    "version": 8,
    "sources": {
        "local": {
            "type": "raster",
            "tiles": [
                "http://localhost:8080/tiles/{z}/{x}/{y}.png"
            ],
            "scheme": "xyz",
            "tileSize": 256
        },
    },
    "layers": [
        {
            "id": "local",
            "type": "raster",
            "source": "local",
            "source-layer": "local",
            "minzoom": 0,
            "maxzoom": 18
        }
    ]
}

r = pdk.Deck(layers=[layer], map_style=custom_map_style, map_provider="mapbox")

Then you can serve your local tiles by running the Python HTTPServer from the parent directory. Below is a simple example that allows CORS requests. This is useful if you export your Deck to a local HTML file like I usually do.

python server.py 127.0.0.1 8080

server.py

from http.server import HTTPServer, SimpleHTTPRequestHandler
import sys

class CORSRequestHandler(SimpleHTTPRequestHandler):

    def end_headers(self):
        self.send_header('Access-Control-Allow-Origin', '*')
        self.send_header('Access-Control-Allow-Methods', '*')
        self.send_header('Access-Control-Allow-Headers', '*')
        self.send_header('Cache-Control', 'no-store, no-cache, must-revalidate')
        return super(CORSRequestHandler, self).end_headers()

    def do_OPTIONS(self):
        self.send_response(200)
        self.end_headers()

host = sys.argv[1] if len(sys.argv) > 2 else '0.0.0.0'
port = int(sys.argv[len(sys.argv) - 1]) if len(sys.argv) > 1 else 8080

print("Listening on {}:{}".format(host, port))
httpd = HTTPServer((host, port), CORSRequestHandler)
httpd.serve_forever()

This image above is a demonstration, actually overlaying a local tile set that I created using datashader, overlaid on tiles provided by OpenStreetMap.

@hokieg3n1us Thank you very much for providing the CORS requests workaround. Worked well for me together with this example from @agressin here: https://github.com/agressin/pydeck_myTileLayer. I really like to try your pydeck code above, to make use of #6121. Like you I have a local tileset, which I have rendered using datashader (following this ). However, definition of layer is missing in your example. So can you please comment on how is layer defined to get OpenStreetMap as the baselayer?

Thanks in advance

[prusswan]

btw, if you are running Jupyter, local files can be accessed over localhost:8888/files/C%3A or localhost:8888/files/%2F (depending on your OS)

wish I had found out earlier