customLoadPath function run per language / namespace to allow caching at asset-level

[dalevfenton]

🚀 Feature Proposal

We serve our language assets as static json files out of a CDN. We want to be able to have the content-hash of each asset used for cache invalidation on an individual asset level. (i.e. we add or update a value in en/common.json it will change the asset name from en/common.abcde.json to en/common.defgh.json, but other asset files like en/login.12434.json will stay the same.

Using the loadPath we are able currently to only set a static hash across all our assets since they all use the same interpolation string (i.e. https://cdn.com/lang/{{lng}}/{{ns}}.qwery.json). We would like to be able through loadPath or another option to return a unique loadPath per language and namespace.

So if loadPath returns a string it would continue to be passed directly to this.services.interpolator.interpolate, but if loadPath returns a map of strings it should try to load each language / namespace combination from the map based on a lookup of language and namespace

Alternatively it could have an extra option called customLoadPaths, that could allow us to pass in something like

{
     "en": { "common": "/lang/{{lng}}/{{ns}}.abcdef.json", "login": "/lang/{{lng}}/{{ns}}.ghjkl.json" },
     "es": { "common": "/lang/{{lng}}/{{ns}}.qwert.json", "login": "/lang/{{lng}}/{{ns}}.yuiop.json" },
}

Motivation

Allow us to setup cache invalidation on a per-asset level instead of having to invalidate all resources if any of them change.

Example

We would pass a config object or function to the options for the plugin that would allow us to specify a custom loadPath per language and namespace. Ideally, loadPath could remain the same and then customLoadPaths could be a function with signature (language: string, namespace: string) => string or a config object: { [K in Languages]?: { [N in Namespaces]?: string } }, if the function or object fails to provide a string for interpolation, then the normal loadPath could be used or the typical error / fallback behavior can be performed

[adrai]

Currently the loadPath option can be a function, so this should already be possible right?

https://github.com/i18next/i18next-http-backend#backend-options

https://github.com/i18next/i18next-http-backend/blob/master/test/http.spec.js#L97

[dalevfenton]

you're right, I thought it only ran this once with all languages and all namespaces, but the arrays are there for multi-loading and it does run the function per-asset needed so it works for what we need

[MetaMmodern]

Hi @dalevfenton , can you please share your loadPath function? I'm interested in how you generate the contenthash, using what. I have a webpack project and was just going to use CopyPlugin to copy translations from public to dist with contenthash, but I don't know how I would get then the same hash in loadPath.

[dalevfenton]

sure @MetaMmodern see below, during local development we load the language files from their location alongside the init code, but when we deploy our apps they use the http backend to fetch the files from a relative path with the other static assets

this is accomplished with the following rule in our webpack prod config which grabs the language files and bundles them into a folder in the build output for deploy, then replaces those reference with a content-hashed value inside the backendResources object

// webpack.prod.js - this is the config that tells webpack how to grab the language files and reference them properly for // the http backend, note that you'll need to make sure your test regex properly matches the path to your language files
module: {
      rules: [
        {
          test: /lang\/[a-z]+\/[a-z]+\.json$/,
          type: 'asset/resource',
          exclude: /node_modules/,
          generator: {
            filename: (resourcePath) => {
              const matches = resourcePath.module.rawRequest.match(/([a-z]+)\/[a-z]+\.json$/);
              const lngPath = matches[1];

              return `lang/${lngPath}/[name].[contenthash:5][ext]`;
            },
          },
        },
      ],
    },

// this is the code for our initI18N, it uses the local language files when in development or tests
// but fetches the assets as an http backend asset when deployed
import enCommon from './lang/en/common.json';
import esCommon from './lang/es/common.json';

const backendResources = {
  resources: {
    en: {
      common: enCommon,
    },
    es: {
      common: esCommon,
    },
  },
};

export const initI18N = async (): Promise<void> => {
  const httpBackendOptions = {
    backend: {
      loadPath: (lngs: string[], nss: string[]) => {
        // since we are not configured for multi-resource lookup, lngs and nss will only have 1 item each
        const lookupLng = lngs?.[0];
        const lookupNs = nss?.[0];

        return backendResources?.resources?.[lookupLng]?.[lookupNs];
      },
    },
  };

  const isDeployed = process.env.DEPLOYED;
  const isTest = process.env.TEST;
  const backendOptions = isTest || !isDeployed ? backendResources : httpBackendOptions;

  // we init i18n with our full list of supported languages but set the language to "en"
  const options = {
    language: 'en',
    fallbackLng: 'en',
    ns: ['common'],
    defaultNS: 'common',
    supportedLngs: ['en', 'es'],
    ...backendOptions,
  };
  let initFn = i18n.use(initReactI18next);

  if (isDeployed) {
    initFn = initFn.use(HttpApi);
  }

  await initFn.init(options);
};

[MetaMmodern]

Very nice, @dalevfenton , thank you After writing to you I wrote a similar code for webpack, but using file-loader and a bit simpler, but that works as well.

But your part with i18n is a bit confusing, you use backendResources as local development, and those contain imported jsons. However, when you're in prod, you have a backend function, loadPath, which grabs resource files by url. How did you tell that function, that your lang resources will have contenthash in the name? Or, how soes i18n understand, that translation files will have contenthash?

I'm gonna make a suggestion: probably the trick is that it gets a list of files from certain language folder, and since you have only one file in that folder you just return it. Is that right?

[dalevfenton]

@MetaMmodern the trick is that the contents of the json files are replace during the build by the webpack asset loader with the value returned by the generator in the loader

as a result after a build it will have a value like below which will match the names of the assets that are hosted in our static directory, the content hash automatically busts the cached asset on the client whenever the contents of those files change based on the content of the source file:

// resulting value during local dev or running tests
const backendResources = {
  resources: {
    en: {
      common: enCommon, // contents of en json file
    },
    es: {
      common: esCommon,  // contents of es json file
    },
  },
};

// resulting value after a production build
const backendResources = {
  resources: {
    en: {
      common: "lang/en/common.b3746.json",
    },
    es: {
      common: "lang/es/common.2800c.json",
    },
  },
};

[MetaMmodern]

@dalevfenton oooh now I see, I wasn't reading carefully, you load THIS when you have prod deployment Amazing, that's exactly what I was trying to achieve today, but the path was not clear. Now I can code with confidence knowing that this is possible and how to do it Thank you!!!