Cached not deleting.

[jhemmmm]

var gRequest = cachedRequest({
            url: photoFetch.data.results,
            ttl: 1000 * 60 * 10 //1 minutes
        }).on("error", function(err) {
            console.log(err);
        });
        gRequest.pipe(res);

Cached file does not seem to be deleting automatically. I am doing it wrong?.

[elhoyos]

Hi @jhemmmm, your code seems ok to me. Notice that at this moment cached-request does not have such a functionality. The cached entries are not removed automatically after they expire, but the request will be made as expected.

We have not considered adding an automated cleanup of expired cache entries, but it sounds useful to have it when setting your cache directory. Something like

cachedRequest.setCacheDirectory(cacheDirectory, {
  cleanupInterval: 24 * 60 * 60 * 1000 // cleanup expired items every 24 hours
});

If you feel like wanting to contribute, feel free to open up a pull request. I would be happy to help you.

[jhemmmm]

I see,

That might not be a good idea on my side. To Identify the expired cache entries, I have to load all saved json files on /tmp/cache. There's at least 2TB of files on my cache directory.

[jhemmmm]

I have tried added a queue job that delete the files base on it's ttl. Here's the forked repo on it. https://github.com/jhemmmm/cached-request/commit/a65b9087aa56cfa9c9e385753d1ce0766ca1a0b5 https://github.com/jhemmmm/cached-request/commit/e40e2869a7e59ca35a1afb5140c42b735c0dc724

It works perfectly fine. Thanks.

EDIT: This thing requires redis. I am still looking for other solution.

[jhemmmm]

I have added a cleanup interval. https://github.com/jhemmmm/cached-request/commit/3cce8e1ff5f070ed7066496cd710ec9f4939bebb https://github.com/jhemmmm/cached-request/commit/5cc08abc1d95fea8766d585c0331a780c7d75d2c https://github.com/jhemmmm/cached-request/commit/24e71a07cd2b6e3ed6c64e94d7658a1267ecfc64

cachedRequest.cleaUpCacheDirectory({
    age: 60 * 60 * 3, // 3 hours
    cleanupInterval: 1000 * 60 * 30 // 30 minutes
});

[elhoyos]

This looks way better without redis. Here are some comments:

• would you mind opening a pull request against this repo, please? This way we could discuss the implementation details there and merge it easily once it is ready.
• I would prefer to make it more performant and reduce the locking the event loop for longer periods with some async find+remove. Anyways, this is a good feature to improve later and I think your solution is good enough for now.
• If there is no strong reason to have cleaUpCacheDirectory[sic], let's set the cleanup options in setCacheDirectory instead.
• let's make the cleanup optional, off by default to not disturb other people using the files.
• age and ttl seem redundant, let's remove it from the options. Starting fresh is an interesting use case you bring to my mind with your age proposal, there you would want to cleanup responses that have not expired yet when re-starting cached-request. That sounds to me like a new feature we could implement in the near future.

Thank you for your ideas and contributions. Please keep them coming.

[jhemmmm]

I used age because the find+remove uses it to check the "creation" of the file. Instead of the ttl given by the cached-request. I'll take a look for it.