Closed broksonic21 closed 8 months ago
Yes I agree, actually I have been thinking of including this in a v4 release that changes up some of basic ways one interacts with the package.
You can do this with a custom cache.
import {
fetchBuilder,
FileSystemCache,
MemoryCache,
getCacheKey,
} from "node-fetch-cache";
import Keyv from "@keyvhq/core";
import KeyvSQLite from "@keyvhq/sqlite";
import { Readable } from "stream";
function streamToBuffer(stream) {
const chunks = [];
return new Promise((resolve, reject) => {
stream.on("data", (chunk) => chunks.push(Buffer.from(chunk)));
stream.on("error", (err) => reject(err));
stream.on("end", () => resolve(Buffer.concat(chunks)));
});
}
const keyvSQLite = new Keyv({
store: new KeyvSQLite("sqlite://fetch_cache_test.sqlite"),
});
class NewCache extends MemoryCache {
constructor(options = {}) {
super({});
this.ttl = options.ttl;
this.cache = keyvSQLite;
this.allowedStatusCodes = [200];
}
async get(key) {
const cachedValue = await this.cache.get(key);
if (cachedValue) {
return {
bodyStream: Readable.from(cachedValue.bodyBuffer),
metaData: cachedValue.metaData,
};
}
return undefined;
}
async remove(key) {
await this.cache.delete(key);
}
async set(key, bodyStream, metaData) {
// bypass to avoid cache non 200 status code
if (!this.allowedStatusCodes.includes(metaData.status))
return { bodyStream, metaData };
const bodyBuffer = await streamToBuffer(bodyStream);
await this.cache.set(key, { bodyBuffer, metaData }, this.ttl);
return this.get(key);
}
}
const fetch = fetchBuilder.withCache(new NewCache());
fetch("https://httpbin.com")
.then((data) => data.text())
.then(console.log);
// additionally you can even delete anything from the cache if you want
// console.log(keyvSQLite.delete(getCacheKey("https://httpbin.com")))
You simply do not save anything to the cache, and here is the result with a 404 page.
Published this as a small package for now. https://www.npmjs.com/package/node-keyv-fetch-cache
It is a small wrapper for keyvhq, so anyone can add anything to the custom cache, as well as ignore adding to cache based on status code.
That's great @entrptaher , thanks for sharing that!
This is also a first class feature now in v4 of node-fetch-cache, check out the control what's cached section in the README. Note that v4 is a major version update and has breaking changes (there's an upgrade guide near the bottom of the README).
I'll close this issue as resolved but feel free to follow-up with any questions or comments.
Been using node-fetch-cache and big fan, but a few pieces of feedback. I'll file them as separate PRs from what we've run into. Thanks for consideration!
I know the docs say you cache all results and user can empty the cache. Be nice if a config option upfront was able to be set for a function or value for what http return codes to cache. Makes a much simpler back and forth in async code.