search

iTowns / itowns

A Three.js-based framework written in Javascript/WebGL for visualizing 3D geospatial data
http://www.itowns-project.org
Other
1.1k stars 298 forks source link

Caching in iTowns #741

Closed zarov closed 6 years ago

zarov commented 6 years ago

I wanted to talk more about caching and how we handle our resources in memory. I know I removed CacheResource in #687, but I'm having some regrets about it, I should have gone further than only replacing it by a Map.

Let's say I fetch a resource, and I want to parse it before using it: the perfect example is with x-bil textures for the elevation. It is logical to add it in the cache after the parsing. But if we are going to make the architecture more malleable, with providers/parsers/others separated, we have a problem here. How can we move the parser outside of this part if we still need to do some cache action after the parsing, knowing the cache is specific to OGCWebServiceHelper here ?

What we need is a global cache, for all classes in iTowns to use (but not in the public API !), as @autra suggested:

  • Provide a cache expiration mechanism, with a configurable duration on a per layer basis? 0 could mean no cache and -1 could mean "cache infinitely".
  • Probably move this behaviour in Scheduler, or at least in a common part, to make every layer benefits of this
  • Maybe use localstorage or something similar to have part of the cache persists across session?

The last point could be taken into account when writing the first Cache, to ease the integration later also with other methods, like the Cache API.

It could currently be used in OGCWebServiceHelper, TileProvider, WFSProvider and 3dTilesProcessing (removing the not so beautiful Math.random mechanism).

Anyone has a thought on this ?

mbredif commented 6 years ago

Just a though, but a complementary strategy would be to limit the cache size by a size in MB rather than by a time expiration. I do not know how easy it is to implement though.

have you looked at https://www.npmjs.com/package/lru-cache ?

zarov commented 6 years ago

I agree, but the size of a javascript object in MB is rather complicated afaik. And the linked package is only having a size in term of number of object in it. I'd rather not add another dependency, but it is subject to discussion what is the best solution !

mbredif commented 6 years ago

about the size, the max cache size in lru-cache is in not necessarily in MB but in the units your length function provides to give an (possibly approximate) size of each object (and defaults to 1, to simply count objects).

That was just a proposition to be considered before we end up recoding it ;). I have no opinion on this maintainance / recoding tradeoff.

zarov commented 6 years ago

Closing since we merged #743