New 'tileserver' binary made with webstor

[klokan]

We need to create a new binary which is going to be distributed together with the MapTiler 0.4 as command line utility and with later version as GUI app - ideally we should split the functionality from interface to be prepared for both interfaces.

The utility will have a name "tileserver" and can be based on the code of the wscmd.

The utility is a remote client for storage of maps in the cloud - it allows initialization of this storage, upload and delate of a map from MBTiles file and destroying of the remote storage.

The basic commands are going to be:

"init" - at given storage (bucket and possibly prefix) creates an empty tileserver.json file containg empty array "[]", later this file should contain list of maps similar to: http://tileserver.maptiler.com/maps.json and inspired by http://mapbox.com/developers/api/. In case provided bucket or prefix does not exist it should ask about creating and create both, if possible. If the file already exist it should give a warning and not overwrite.

"info" - will print the list of maps loaded from tileserver.json remote endpoint (bucket and possibly prefix)

"add" - will upload tiles from a specified mbtiles file with XYZ scheme and metadata.json file.

"remove" - will efficiently delete a map from remote storage (record from the tileserver.json and all tiles and metadata.json).

"destroy" - will delete the whole remote tileserver instance (including buckets) - after confirmation.

The first version must support at least Amazon S3 and (if possible also) Google Cloud Storage.

The remote endpoint should be preferably specified as one string, such as: s3://bucket-name/directory/ or just s3://bucket to simplify the parameters

[xrosecky]

OK, I assume that username and password will be stored in a file. The commandline will share the same interface with progress bar support as the later GUI app.

For Google Cloud Storage I would use this method for authentication/authorization: https://developers.google.com/storage/docs/accesscontrol#Signed-URLs

Later we can also implement authentication/authorization using OAuth 2.0 protocol.

[xrosecky]

I created tileserver branch and added the interface for tileserver.

[xrosecky]

Implementation of command "add" is finished, "init" is in progress.

@klokan: I have a question regarding the tileserver.json file. For example I have this URL for "add" operation:

s3://bucket-name/directory/grandcanyon

Where tileserver.json will be stored? In s3://bucket-name/directory/tileserver.json - the last part of the URL will be removed or it should be specified by user?

Do you have some preference for JSON library for C/C++? There is an overview of JSON libraries on http://www.json.org/ and for example http://lloyd.github.io/yajl/ seems interesting (small, portable, reasonable license).

[klokan]

Great, thanks.

The URL is the url to the tileserver.json location, it should not contain the name of the map ("grandcanyon"). 'Add' should complain if the tileserver.json is not found on that location. You can't 'add' a map without proper 'init' before! The 'init' may create a bucket with give name if it does not exists, and it must create the tileserver.json file.

I expect the typical URL used is going to be just s3://bucket-name (maybe we can even enforce it and disallow directory names, but it would be better to support directory or prefix in in the implementation as well).

Re the JSON library. We need something what allows parsing of relatively small JSON into memory, some modification (adding a new map into list on given position), serialising back to JSON. I am in favour of: https://github.com/kbranigan/cJSON

[xrosecky]

OK, I will use cJSON library.

[xrosecky]

Which version of TileJSON do you want to use?

[klokan]

I need to create a metadata.json file - which contains everything what the mbtiles has in the metadata table, just serialized into JSON format, exactly as in this file: {{{ { "profile": "mercator", "name": "grandcanyon tiles", "format": "png", "bounds": "-112.26379395,35.98245136,-112.10998535,36.13343831", "minzoom": "10", "version": "1.0.0", "maxzoom": "16", "type": "overlay", "description": "" } }}} The mbutil (https://github.com/mapbox/mbutil) utility is exporting from mbtiles and importing into mbtiles exactly the same format. You can't generate a correct TileJSON endpoint, becuase JSONP can't be done in static files in Amazon S3 and because you don't know all the information at the time of upload (such as all available CNAME for the amazon bucket).

[xrosecky]

OK, I interchanged tileserver.json with TileJSON.

[klokan]

tileserver.json is just list of maps. Similar to: http://tileserver.maptiler.com/maps.json, but simpler.

In the first version we need attributes: "basename" (jmeno mbtiles souboru bez pripony), "scheme" (vzdy xyz), "profile" (vzdy 'mercator'), bounds (tady asi jako list 4 cisel), minzoom, maxzoom.

Jakmile bude prvni verze s timhle - dospecifikuju presne detaily.

[xrosecky]

OK, tileserver.json now contains (values are taken from metadata table):

[
  {
    "name":"brno",
    "type":"overlay",
    "version":"1.0.0",
    "description":"",
    "format":"png",
    "minzoom":"-140165568",
    "maxzoom":"32625",
    "bounds":"-112.26379395,35.98245136,-112.10998535,36.13343831"
  },
  {
    "name":"grandcanyon",
    "type":"overlay",
    "version":"1.0.0",
    "description":"",
    "format":"png",
    "minzoom":"10",
    "maxzoom":"16",
    "bounds":"-112.26379395,35.98245136,-112.10998535,36.13343831"
  }
]

So I will add basename, scheme and profile and bounds will be array of integers instead of string.

[xrosecky]

Attributes basename, scheme, profile and bounds added to tileserver.json and metadata.json. There is tileserver.json example:

[
  {
    "basename":"test2",
    "name":"brno",
    "type":"overlay",
    "version":"1.0.0",
    "description":"",
    "format":"png",
    "minzoom":"10",
    "maxzoom":"16",
    "scheme":"xyz",
    "profile":"mercator",
    "bounds":[-112.264000, 35.982500, -112.110000, 36.133400]
  },
  {
    "basename":"test1",
    "name":"grandcanyon",
    "type":"overlay",
    "version":"1.0.0",
    "description":"",
    "format":"png",
    "minzoom":"10",
    "maxzoom":"16",
    "scheme":"xyz",
    "profile":"mercator",
    "bounds":[-112.264000, 35.982500, -112.110000, 36.133400]
  }
]

[xrosecky]

How do you want to implement destroy method? Now destroy operation check for tileserver.json and if it exists, then deletes all including content, that was not uploaded by tileserver (eg. some content inserted by the user itself, which share the same prefix). The alternative is to parse tileserver.json and give user a prompt, if there are files not referenced from tileserver.json?

Do you have some preffered method how to store login information (username, password) - now they are specified on commandline (not secure, at least on UNIX OS). Simple ini file like this?

username=testuser
password=testpwd

[klokan]

Deleting of everything is fine - we should ask before it is done. No need for external storage of password or login - the system variable and/or parameter on the command line is fine. Thanks.

[xrosecky]

OK, tileserver binary is ready for testing.

[MartinMikita]

@xrosecky there is a mistake in the help, see the following lines:

'info'     print the list of maps loaded from tileserver.json remote endpoint:
                  tileserver add url username password mbtile

And another text mistake:

./tileserver list s3://webstortest6 - -
Unknow operation given, supported operations are add.

[xrosecky]

OK, I will update the help. Should I use tileserver-maptiler branch?

[MartinMikita]

Yes, it would be great. Thanks.

[klokan]

@xrosecky Vasek, could you please also update the template for generated index.html - so that it contains:

...
  <link rel="stylesheet" type="text/css" href="https://maptilercdn.s3.amazonaws.com/tileserver.css" />
  <script src="https://maptilercdn.s3.amazonaws.com/tileserver.js"></script>
</head>
<body>
<script>tileserver();</script>
...

The JavaScript is now ready - and will automatically request tileserver.json via AJAX and create an clickable web interface - same as at: http://tileserver.maptiler.com/.

[klokan]

Please use https to load the CSS and JS - I have updated the above block of code already.

[xrosecky]

OK, index.html is updated and I also fixed the help.