search

canjs / can-memory-store

Create an in-memory datastore that supports mongo-like queries
https://canjs.com/doc/can-memory-store.html
MIT License
2 stars 0 forks source link

cache lifetime ~21 #3

Open justinbmeyer opened 6 years ago

justinbmeyer commented 6 years ago

@pYr0x commented on Sun Nov 22 2015

with data-localstorage-cache data will be cached in the browsers localstorage.

if a user visit the website, can-connect will return immediately the localstroage data (if present) like the video explain: fall-through-cache

to clear the cache, we can use: http://connect.canjs.com/doc/can-connect|data|localstorage-cache.clear.html

but sometimes, it is useful to have a cache lifetime.

real world example: if you have a news webpage, and the user visit your website at the first time, there is no data on the localstorage, so the promise is pending until it is resolved and the news are displayed next day, the user visit your website again. a lot of news has changed. the user will first get the old news, after the promise is resolved, the new news will be displayed.

@justinbmeyer commented on Sun Nov 22 2015

This type of strategy can be added by a user, but probably won't go directly into a cache behavior. Instead a different behavior should be created that can manage what is in the cache.

Sent from my iPhone

On Nov 22, 2015, at 7:51 AM, pYr0x notifications@github.com wrote:

with data-localstorage-cache data will be cached in the browsers localstorage.

if a user visit the website, can-connect will return immediately the localstroage data (if present) like the video explain:

to clear the cache, we can use: http://connect.canjs.com/doc/can-connect|data|localstorage-cache.clear.html

but sometimes, it is useful to have a cache lifetime.

real world example: if you have a news webpage, and the user visit your website at the first time, there is no data on the localstorage, so the promise is pending until it is resolved and the news are displayed next day, the user visit your website again. a lot of news has changed. the user will first get the old news, after the promise is resolved, the new news will be displayed.

— Reply to this email directly or view it on GitHub.

@pYr0x commented on Mon Nov 23 2015

i think it belongs to a cache strategy to manage how long the cache would be exit.

i thinks is very easy to implement this in localstorage-cache. and for other people it will be very easy to use this without creating their own cache behavior. creating a own cache behavior can be very trick... (for me)

@justinbmeyer commented on Mon Nov 23 2015

@pYr0x But memory-cache might want it too. Same with an indexDB storage. How about creating a behavior that can be added to either localstorage or memory?

I've also wanted an LRU behavior as well.

The problem is we'd want to be able to store a date or some other bit of data associated with a set. For example, in localStorage, instead of storing a set like:

"todos/set/{}": [1,2,3]

It will have to support metadata like:

"todos/set/{}": {data: [1,2,3], lastUpdated: 3452525524}

Fortunately, metadata can already exist on response data like:

{
  data: [{...}, ....],
  count: 100,
  otherMetaData: 20
}

So, I think there are a few things that need to happen:

  1. localstorage and memory caches need to support metadata.
  2. a lifetime-cache behavior should be created that wraps localstorage or memory. It adds a lastUpdated timestamp to everything and uses it accordingly.

This allows creating a cache connection like:

connect(['data-localstorage-cache','lifetime-cache',{
  name: "todos",
  lifetime: 1000*60*60*24 // 1 day
})

Sudo code:

module.exports = connect.behavior("lifetime-cache",function(baseConnect){
  getListData: function(set){
    set = set || {};
    var lifetime = this.lifetime;
    return new Promise(function(){
      baseConnect.getData(set).then(function(data){
        if(data.lastUpdated >= new Date() - lifetime) {
          resolve(data)
        } else {
          // removeSet
          reject({message: "no data", error: 404})
        }
      }, reject);
    })

  }
});

@pYr0x commented on Mon Nov 23 2015

i like your idea of a new behavior "lifetime-cache" thats wrapps the cache.

i won't add a lastUpdated to a set. if we delete only some sets this can make strange sorting behaviors in stache templates when using a {{#each}} list

i suggest to add a date at the cache (new Date + lifetime). if the date is in the past, delete the whole cache, provided by the cacheConnection.

localstorage: todosCache-expires: 2015-11-23T17:59:00

@pYr0x commented on Mon Nov 23 2015

hey justin,

what about that?

module.exports = connect.behavior('cache-lifetime', function (baseConnect) {
        var behavior = {
            getListData: function (set) {
                debugger;
                set = set || {};
                var self = this;

                var current = new Date();
                var expires = new Date(localStorage.getItem(self.cacheConnection.name + '-expires'));

                if(isNaN(expires) || expires.getTime() < current.getTime()) {
                    this.cacheConnection.clear();
                }
                return this.cacheConnection.getListData(set).then(function (data) {
                    self._getHydrateList(set, function (list) {
                        self.addListReference(list, set);
                        setTimeout(function () {
                            baseConnect.getListData.call(self, set).then(function (listData) {
                                self.updateListData(listData, set);
                                self.updatedList(list, listData, set);
                                self.deleteListReference(list, set);
                            }, function (e) {
                                console.log('REJECTED', e);
                            });
                        }, 1);
                    });
                    return data;
                }, function () {
                    var listData = baseConnect.getListData.call(self, set);
                    listData.then(function (listData) {
                        self.updateListData(listData, set);
                    });
                    return listData;
                });
            },
            updateListData: function(listData, set) {
                var current = new Date();
                var expires = new Date(current.getTime() + this.cacheLifetime * 60000);
                localStorage.setItem(this.cacheConnection.name + '-expires', expires);
                this.cacheConnection.updateListData(listData, set);
            }
        };
        return behavior;
    });

i added the cache-lifetime before inline-cache and before fall-through-cache

connect.order = [
        'data-localstorage-cache',
        'data-url',
        'data-parse',
        'cache-requests',
        'data-combine-requests',
        'constructor',
        'constructor-store',
        'can-map',
        'fall-through-cache',
        'data-inline-cache',
        'cache-lifetime',
        'data-worker',
        'data-callbacks-cache',
        'data-callbacks',
        'constructor-callbacks-once'
    ];

maybe not the best code, but trend? i have to repleat the fall-through-cache getListData-function almost complete, maybe you have a better solution

@justinbmeyer commented on Mon Nov 23 2015

i won't add a lastUpdated to a set. if we delete only some sets this can make strange sorting behaviors in stache templates when using a {{#each}} list

I don't understand. When you {{#each list}} a list retrieved by .getList ... it will either be in the cache or it wont.

I'm not sure what strange sorting behavior could happen?

@justinbmeyer commented on Mon Nov 23 2015

I would prefer to have the lifetime be associated with different requests. Say I retrieved:

// Three weeks ago 
.getList({type: 'spam'})

// 1 hr ago
.getList({type: 'inbox'})

If I make a request for .getList({type: 'inbox'}) that should be a cache hit, but .getList({type: 'spam'}) should not.

@pYr0x commented on Mon Nov 23 2015

lets say we have 3 sets in localstorage:

  1. all
  2. complete
  3. open

lets say we have 5 items, 3 open, 2 complete the order of displaying them is the following

  1. [open item]
  2. [complete item]
  3. [open item]
  4. [complete item]
  5. [open item]

if we call .getList({group: open}) and .getList({group: complete}). the first set .getList({}) contain them all.

if we have separate lifetimes for each set, what would be happen if .getList({group: complete}) has a new expiring date as .getList({group: open}). so set {group: complete} will be removed and updated after the promise is resolved

i expect, the list would look like

  1. [open item]
  2. [open item]
  3. [open item]

and afer the promise is resolved

  1. [open item]
  2. [open item]
  3. [open item]
  4. [complete item]
  5. [complete item]

am I wrong?

@justinbmeyer commented on Mon Nov 23 2015

I don't think you would have those three sets in localStorage. They complete and open would get combined into all.

@justinbmeyer commented on Mon Nov 23 2015

if we have separate lifetimes for each set, what would be happen if .getList({group: complete}) has a new expiring date as .getList({group: open})

I don't know what you mean by: "has a new expiring date as".

@justinbmeyer commented on Mon Nov 23 2015

Also, by openand complete are these different and independent values of group? A property that would be defined with set.comparators.enum?

@justinbmeyer commented on Mon Nov 23 2015

In general, you bring up a good point ... what happens with the timestamp with subsets? I think it would be ok to start to simply have the superset's timestamp win.