search

jones139 / BrewMap

The client and server side code to produce a map of uk breweries based on OpenStreetMap Data (http://wiki.openstreetmap.org/wiki/WikiProject_United_Kingdom_Breweries)
http://brewmap.maps3.org.uk
6 stars 3 forks source link

Improve Scalability of Server Side Code #27

Open jones139 opened 12 years ago

jones139 commented 12 years ago

The current approach of extracting all of the data into a single file is ok for the UK drinks industry, but will not scale well to either more dense points of interest, or a larger area.

The client should be able to request the features for a specific area as the map is panned and zoomed.

There are a few ways this could be done - will start to list them later.....

jones139 commented 12 years ago

I think the options are:

  1. The current way - maintain an osm2pgsql database on a database server, and keep it up to date using osmosis and osm2pgsql. Generate the data files periodically and up-load them to web server. Pros: Simple, if database server crashes, web front end continues to work using latest available data. Cons: Does not scale well - one file per layer would get big if we try to run map for larger area, or for higher density features.
  2. Use Mapnik2 'MetaWriter' to generate image tiles for the map, but provide a data file for each tile to provide the interactive part of the map (data in popups). Use tilecache to upload tiles and metadata from database server to web server. Pros: Server side simple for static data - will need work to deal with expiry of tiles and meta-data. Cons: I think we will need a Leaflet plug-in to retrieve the meta-data each time a tile is downloaded, then add the data to the icon layer groups.
  3. Run an instance of xapi and have Leaflet generate an xapi query each time a tile is loaded to obtain the information. Pros: jxapi will be supported so should be able to get it running. Cons: Can not run jxapi on web server, so will be limited by upload speed of my internet connection to the database server, so could seem slow from a users point of view.
  4. Use database server to obtain diffs using osmosis and generate sql to keep a mysql database up to date on the web server. Pros - should be fast for the users because it uses the mysql server on the web server. Cons - will need a Leaflet plugin to obtain the data as tiles are loaded. Will need to develop the code to keep the mysql database up to date.

I think the options for scalability using our current hardware are 2 (mapnik metawriters) or 4 (mysql database on web server). I would quite like to get the mysql version working, but the mapnik route will be the easiest. What do you think?

jones139 commented 12 years ago

I am minded to have a go at mapnik metawriters (http://sotm-eu.org/slides/58_HermannKraus_vortrag.pdf). I could make a set of transparent tiles with just small icons for the BrewMap features, and json files corresponding to each map tile. Then it is just a matter of getting the javascript side working. Will give that a bit of thought tomorrow evening.

jones139 commented 12 years ago

Actually, I have been thinking some more - we do not actually need raster tiles produced by mapnik, so an Option 5 would be to use TileCache (http://tilecache.org) with a new backend - rather than a module that calls mapnik to produce a png of a particular area, it could query the database and produce a json file for a particular area - the json files would then be cached so they are quick to access next time. Still need the same client code to download the json each time a tile is downloaded and add the data to the map.