search

i18next / i18next-parser

Parse your code to extract translation keys/values and manage your catalog files
MIT License
466 stars 189 forks source link

s.codePointAt is not a function #993

Open dobesv opened 3 months ago

dobesv commented 3 months ago

🐛 Bug Report

When using the transform to process files, I get a stack trace:

 s.codePointAt is not a function
    TypeError: s.codePointAt is not a function
        at codePointAt (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:12414:81)
        at Object.scan (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:11509:26)
        at nextTokenWithoutCheck (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:32756:43)
        at nextToken (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:32768:20)
        at parseSourceFileWorker (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:32443:13)
        at Object.parseSourceFile (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:32278:26)
        at Object.createSourceFile (xxx/.yarn/cache/typescript-patch-32ada147aa-5659316360.zip/node_modules/typescript/lib/typescript.js:32078:29)
        at JavascriptLexer.extract (xxx/.yarn/cache/i18next-parser-npm-5.4.0-759fe94453-0e8a23b28b.zip/node_modules/i18next-parser/dist/lexers/javascript-lexer.js:77:27)
        at Parser.parse (xxx/.yarn/cache/i18next-parser-npm-5.4.0-759fe94453-0e8a23b28b.zip/node_modules/i18next-parser/dist/parser.js:75:36)
        at i18nTransform._transform (xxx/.yarn/cache/i18next-parser-npm-5.4.0-759fe94453-0e8a23b28b.zip/node_modules/i18next-parser/dist/transform.js:107:33)
        at i18nTransform.Transform._write (node:internal/streams/transform:171:8)
        at writeOrBuffer (node:internal/streams/writable:564:12)
        at _write (node:internal/streams/writable:493:10)
        at i18nTransform.Writable.write (node:internal/streams/writable:502:10)
        at Readable.ondata (node:internal/streams/readable:1007:22)
        at Readable.emit (node:events:518:28)

This error does NOT occur in node 20.10.0 but does occur in node 20.11.0 onwards. I think they changed something around streams and encoding.

When I debug more deeply I found that when i18next-parser reads the file, in the newer version of node it is getting back a buffer because it passes in undefined for the encoding.

To Reproduce

This the code that fails starting in node v20.11.0:

const parseFiles = async (files: string[]) => {
  const i18nextTransform = require('i18next-parser').transform;
  const transform = new i18nextTransform({
    contextSeparator: '_',
    createOldCatalogs: false,
    defaultNamespace: 'translation',
    defaultValue: '',
    indentation: 0,
    keepRemoved: false,
    keySeparator: '.',
    lexers: {
      js: ['JavascriptLexer'],
      ts: ['JavascriptLexer'],
      jsx: [
        {
          lexer: 'JsxLexer',
          transSupportBasicHtmlNodes: true,
          attr: 'i18nKey',
        },
      ],
      tsx: [
        {
          lexer: 'JsxLexer',
          transSupportBasicHtmlNodes: true,
          attr: 'i18nKey',
        },
      ],
      default: ['JavascriptLexer'],
    },
    lineEnding: 'auto',
    locales: ['en'],
    namespaceSeparator: ':',
    pluralSeparator: '_',
    output: 'locales/$LOCALE/$NAMESPACE.json',
    sort: true,
    useKeysAsDefaultValue: false,
    skipDefaultValues: false,
    failOnWarnings: false,
  });

  const content: Array<{
    namespace: string;
    content: Record<string, unknown>;
  }> = [];

  await once(
    Readable.from(files.map(path => ({ isBuffer: () => false, path })))
      .pipe(transform)
      .on('data', (file: { contents: string; path: string }) => {
        content.push({
          namespace: path.basename(file.path, '.json'),
          content: JSON.parse(file.contents),
        });
      })
      .on('warning', (warning: string) => warnings.push(warning)),
    'finish',
  );

  return content;
};

Workaround

I can workaround the issue by loading the file content myself and passing the contents as a buffer:

  await once(
    Readable.from(
      files.map(path => ({
        isBuffer: () => true,
        path,
        contents: readFileSync(path),
      })),
    )
      .pipe(transform)
      .on('data', (file: { contents: string; path: string }) => {
        content.push({
          namespace: path.basename(file.path, '.json'),
          content: JSON.parse(file.contents),
        });
      })
      .on('warning', (warning: string) => warnings.push(warning)),
    'finish',
  );

Expected behavior

It should work equally well whether in node v20.10.0, v20.11.0, or v20.12.0

Your Environment