Support not caching for non-200s

[broksonic21]

Been using node-fetch-cache and big fan, but a few pieces of feedback. I'll file them as separate PRs from what we've run into. Thanks for consideration!

I know the docs say you cache all results and user can empty the cache. Be nice if a config option upfront was able to be set for a function or value for what http return codes to cache. Makes a much simpler back and forth in async code.

[mistval]

Yes I agree, actually I have been thinking of including this in a v4 release that changes up some of basic ways one interacts with the package.

[entrptaher]

You can do this with a custom cache.

import {
  fetchBuilder,
  FileSystemCache,
  MemoryCache,
  getCacheKey,
} from "node-fetch-cache";
import Keyv from "@keyvhq/core";
import KeyvSQLite from "@keyvhq/sqlite";
import { Readable } from "stream";

function streamToBuffer(stream) {
  const chunks = [];
  return new Promise((resolve, reject) => {
    stream.on("data", (chunk) => chunks.push(Buffer.from(chunk)));
    stream.on("error", (err) => reject(err));
    stream.on("end", () => resolve(Buffer.concat(chunks)));
  });
}

const keyvSQLite = new Keyv({
  store: new KeyvSQLite("sqlite://fetch_cache_test.sqlite"),
});

class NewCache extends MemoryCache {
  constructor(options = {}) {
    super({});
    this.ttl = options.ttl;
    this.cache = keyvSQLite;
    this.allowedStatusCodes = [200];
  }

  async get(key) {
    const cachedValue = await this.cache.get(key);
    if (cachedValue) {
      return {
        bodyStream: Readable.from(cachedValue.bodyBuffer),
        metaData: cachedValue.metaData,
      };
    }

    return undefined;
  }

  async remove(key) {
    await this.cache.delete(key);
  }

  async set(key, bodyStream, metaData) {
    // bypass to avoid cache non 200 status code
    if (!this.allowedStatusCodes.includes(metaData.status))
      return { bodyStream, metaData };

    const bodyBuffer = await streamToBuffer(bodyStream);
    await this.cache.set(key, { bodyBuffer, metaData }, this.ttl);
    return this.get(key);
  }
}

const fetch = fetchBuilder.withCache(new NewCache());

fetch("https://httpbin.com")
  .then((data) => data.text())
  .then(console.log);

// additionally you can even delete anything from the cache if you want
// console.log(keyvSQLite.delete(getCacheKey("https://httpbin.com")))

You simply do not save anything to the cache, and here is the result with a 404 page.

[entrptaher]

Published this as a small package for now. https://www.npmjs.com/package/node-keyv-fetch-cache

It is a small wrapper for keyvhq, so anyone can add anything to the custom cache, as well as ignore adding to cache based on status code.

[mistval]

That's great @entrptaher , thanks for sharing that!

This is also a first class feature now in v4 of node-fetch-cache, check out the control what's cached section in the README. Note that v4 is a major version update and has breaking changes (there's an upgrade guide near the bottom of the README).

I'll close this issue as resolved but feel free to follow-up with any questions or comments.