search

skeate / Leaflet.timeline

Display arbitrary GeoJSON on a map with a timeline slider and play button
https://skeate.github.io/Leaflet.timeline
ISC License
391 stars 67 forks source link

Database "caching" of GeoJSON layers #25

Open DrYSG opened 8 years ago

DrYSG commented 8 years ago

With the ability to handle multiple GeoJSON layers in the Timeline (https://github.com/skeate/Leaflet.timeline/issues/23) comes the risk of running out of memory by keeping all data in core.

I did some work a year+ ago on using PouchDB (layers over indexdb) to cache raster tiles. http://gis.stackexchange.com/questions/62070/offline-slippy-map-tiles-database-for-leaflet.

@calvinmetcalf has done some great work on serving GeoJSON from PouchDB (https://github.com/calvinmetcalf/leaflet.pouch)

Now we have the refactoring and separation of the Timeline from the GeoJSON layers (https://github.com/skeate/Leaflet.timeline/issues/23). What about being able to use a PouchDB back-end DB for dealing with 20MB+ (I have gone as big as 1GB for my PouchDB raster work). My experience has been that serving from the browser-internal database is pretty fast. There is lots of locality of reference.

One approach would be to extend the leaflet.GeoJSON.PouchDB plugin with an additional index of the Date field, and do searches for start>date>end as map based query indexes. The trick is that both Tileline and Leaflet.GeoJSON.Pouch are peer sub-classes of GeoJSON layer.

Perhaps easier would be to just have option on Leaflet.Timeliner to have a method called:

fetchGeoJSON(layer, start, end)

which which would be fired to load the GeoJSON data on demand. That would leave it to the user to manage the Database. I worked with a similar (very short, and cute) thing called functional.tilelayer by @ismyrnow https://github.com/ismyrnow/Leaflet.functionaltilelayer that did async calls to a DB to return tiles. In this case it would return GeoJSON, not tiles. The advantage here is that one could also invoke remote calls instead of just to the local DB.

I don't think we would have to go to GeoJSON tiles for now, but that is also a possibility for some.

skeate commented 8 years ago

At first I was thinking I might be able to do something neat with prototypal inheritance (basically, dynamically change the parent class of L.Timeline from L.GeoJSON to L.GeoJSON.Pouch), but I think I'd wind up overriding almost all of what leaflet.pouch does anyway.

Will look at adding it, though. I can see the utility; the country border example's json is about 20MB, and it's slow to load at first. Would be neat to essentially stream that data in on demand.

calvinmetcalf commented 8 years ago

you might be better off using topojson and having a different object for each time period, you could convert to geojson on demand and any duplicate geometry would get deduped

On Thu, Dec 3, 2015 at 12:39 PM Jonathan Skeate notifications@github.com wrote:

At first I was thinking I might be able to do something neat with prototypal inheritance (basically, dynamically change the parent class of L.Timeline from L.GeoJSON to L.GeoJSON.Pouch), but I think I'd wind up overriding almost all of what leaflet.pouch does anyway.

Will look at adding it, though. I can see the utility; the country border example's json is about 20MB, and it's slow to load at first. Would be neat to essentially stream that data in on demand.

— Reply to this email directly or view it on GitHub https://github.com/skeate/Leaflet.timeline/issues/25#issuecomment-161726597 .

DrYSG commented 8 years ago

Well @calvinmetcalf , TopoJSON would not solve my needs. I am dealing with sensor data that creates footprints and the footprint changes every 10ms to 1000 ms. So there is very little in the way of common arcs for TopoJSON.

What I was hoping to get from you was some deep insight from someone connecting to PouchDB and especially your plug-in.

  1. Does your plug-in make use use indexes for lat/long ? 2, Would adding a new one for time permit faster access
  2. Given that this pretty much read-only data, the CouchDB paradigm of taking a hit on the first reference make sense
  3. Or should we nix the idea of trying to build on the Leaflet.GeoJSON.Pouch plug-in and just store the data directly into a PouchDB given that we are interested mainly in time locality?
  4. Any other issues or concerns to think about?
  5. Would Dale or Nolan find this interesting?