Bug: Cannot extract keys from built-in control flow blocks

austinw-fineart commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Is this a regression?

No

Current behavior

Angular v17 introduces a new built-in control flow syntax that isn't recognized by the keys extractor.

Expected behavior

This simple block should work:

<ng-container *transloco="let t">
  @if (a > b) {
    {{ t('a is greater than b') }}
  }
</ng-container>

Please provide a link to a minimal reproduction of the bug

N/A

Transloco Config

No response

Debug Logs

No response

Please provide the environment you discovered this bug in

Transloco: 5.0.7
Transloco Keys Manager: 3.8.0
Angular: 17.0.2
Node: 20.9.0
Package Manager: npm 10.2.3
OS: Windows 11 (22H2)

Additional context

No response

I would like to make a pull request for this bug

No

Celtian commented 1 year ago

I agree. This is crucial and need to be done ASAP. Otherwise I will do my own extractor...

shaharkazaz commented 1 year ago

@Celtian You are more than welcome to open a PR 👍 I won't be available in the near future as my country is at war. This upgrade is something I started on v16 as well but it's a very big change as it requires to move to ESM package

tleveque23 commented 1 year ago

@shaharkazaz This is a pretty big issue. Is there anybody other than you, that can work on that?

shaharkazaz commented 1 year ago

@tleveque23 This is an open source, anyone from the community is welcome to contribute.

Celtian commented 1 year ago

I have already made some script. Probably this is not as powerful as this library, but I hope it helps as temporary solution.

/* eslint-disable @typescript-eslint/no-explicit-any */
import { readFileSync, writeFileSync } from 'fs-extra';
import { globStream } from 'glob';
import * as path from 'path';

interface AbstractUpdateLangConfig {
  encoding: BufferEncoding;
  defaultValue: string;
}

interface AbstractKeyStoreConfig {
  keyStore: Set<string>;
}

interface AbstractCwdConfig {
  cwd: string;
}

interface AbstractLangsConfig {
  langs: string | string[];
}

interface FindOccuranceConfig extends AbstractKeyStoreConfig {
  regex: RegExp[];
  fileContent: string;
}

interface UpdateLangConfig extends AbstractUpdateLangConfig, AbstractKeyStoreConfig {
  langPath: string;
}

interface UpdateLangsConfig
  extends AbstractUpdateLangConfig,
    AbstractKeyStoreConfig,
    AbstractCwdConfig,
    AbstractLangsConfig {}

interface ParserConfig {
  formula: (key: string) => string;
  type: 'single' | 'double' | 'both';
  coveredCases: string[];
}

interface MainConfig extends AbstractUpdateLangConfig, AbstractCwdConfig, AbstractLangsConfig {
  dryRun: boolean;
  source: string | string[];
  regex: {
    html: ParserConfig[];
    typescript: ParserConfig[];
  };
}

const flattenJson = (obj: any, parentKey: string = '', separator: string = '.'): Record<string, string> => {
  let result: Record<string, string> = {};

  for (const key in obj) {
    if (obj.hasOwnProperty(key)) {
      const newKey = parentKey ? `${parentKey}${separator}${key}` : key;

      if (typeof obj[key] === 'object' && obj[key] !== null) {
        const flattenedSubObj = flattenJson(obj[key] as any, newKey, separator);
        result = { ...result, ...flattenedSubObj };
      } else {
        result[newKey] = obj[key] as string;
      }
    }
  }

  return result;
};

const findOccurance = (config: FindOccuranceConfig): void => {
  for (const rx of config.regex) {
    let matchesMarker;
    while ((matchesMarker = rx.exec(config.fileContent)) !== null) {
      const key = matchesMarker.groups?.['content']?.trim();
      if (key) {
        config.keyStore.add(key);
      }
    }
  }
};

const updateLang = (config: UpdateLangConfig): void => {
  const langFileContent = readFileSync(config.langPath).toString();

  const flattenedLang = flattenJson(JSON.parse(langFileContent));

  const langKeyStore = new Map<string, string>();

  for (const [k, v] of Object.entries(flattenedLang)) {
    langKeyStore.set(k, v);
  }

  const result: Record<string, any> = {};

  for (const key of Array.from(config.keyStore).sort((a, b) => a.localeCompare(b, 'en'))) {
    const keys = key.split('.');
    let currentObj: Record<string, any> = result;

    for (let i = 0; i < keys.length; i++) {
      const currentKey = keys[i];
      currentObj = currentObj[currentKey] =
        currentObj[currentKey] || (i === keys.length - 1 ? langKeyStore.get(key) || config.defaultValue : {});
    }
  }

  writeFileSync(config.langPath, JSON.stringify(result, null, 2), {
    encoding: config.encoding,
  });
};

const updateLangs = (config: UpdateLangsConfig): void => {
  console.log('\n✔ Writing result into language files');

  const files: string[] = [];

  const langFilesStream = globStream(config.langs, {
    cwd: config.cwd,
  });

  langFilesStream.on('data', async (langPath) => {
    files.push(langPath);

    updateLang({
      defaultValue: config.defaultValue,
      encoding: config.encoding,
      keyStore: config.keyStore,
      langPath: path.join(config.cwd, langPath),
    });
  });

  langFilesStream.on('end', () => {
    console.log(`ℹ Keys were were updated in:\n`);

    console.table(files);

    console.log('\n🌵 Done! 🌵\n');
  });
};

const main = async (config: MainConfig): Promise<void> => {
  console.log('Starting Translation Files Build 👷🏗');

  console.log('\n✔ Extracting Template and Component Keys 🗝');

  const keyStore = new Set<string>();
  let filesCount = 0;

  const rxSingleQuotes = /'(?<content>([^'\s]|\\')+)'/;
  const rxDoubleQuotes = /"(?<content>([^"\s]|\\")+)"/;

  const filesStream = globStream(config.source, {
    cwd: config.cwd,
  });

  filesStream.on('data', async (filePath) => {
    filesCount++;
    const fileContent = readFileSync(path.join(config.cwd, filePath), {
      encoding: config.encoding,
    }).toString();

    const createRegex = (parser: ParserConfig): RegExp[] => {
      if (parser.type === 'single') {
        return [new RegExp(parser.formula(rxSingleQuotes.source), 'g')];
      } else if (parser.type === 'double') {
        return [new RegExp(parser.formula(rxDoubleQuotes.source), 'g')];
      }
      return [
        new RegExp(parser.formula(rxSingleQuotes.source), 'g'),
        new RegExp(parser.formula(rxDoubleQuotes.source), 'g'),
      ];
    };

    if (filePath.endsWith('.ts')) {
      findOccurance({
        regex: [...config.regex.typescript.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    } else if (filePath.endsWith('.html')) {
      findOccurance({
        regex: [...config.regex.html.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    }
  });

  filesStream.on('end', () => {
    console.log(`ℹ ${keyStore.size} keys were found in ${filesCount} files.`);

    if (!config.dryRun) {
      updateLangs({
        cwd: config.cwd,
        defaultValue: config.defaultValue,
        encoding: config.encoding,
        langs: config.langs,
        keyStore,
      });
    } else {
      console.log(`ℹ Dry run activated. Language files will not be updated.`);

      console.log('\n🌵 Done! 🌵\n');
    }
  });
};

main({
  dryRun: false,
  encoding: 'utf-8',
  defaultValue: '███',
  cwd: path.join(process.cwd(), '..'),
  source: ['projects/portal/src/app/**/*.ts', 'projects/portal/src/app/**/*.html'],
  langs: ['projects/portal/src/assets/i18n/*.json'],
  regex: {
    html: [
      {
        formula: (key: string): string => `{{\\s*${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?}}`,
        type: 'both',
        coveredCases: [
          `{{ 'uni.close' | transloco }}`,
          `{{ "uni.close" | transloco }}`,
          `{{ 'uni.close' | transloco: variable }}`,
          `{{ "uni.close" | transloco: variable }}`,
        ],
      },
      {
        formula: (key: string): string => `"${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?"`,
        type: 'single',
        coveredCases: [`"'uni.close' | transloco"`, `"'uni.close' | transloco : variable"`],
      },
      {
        formula: (key: string): string => `'${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?'`,
        type: 'double',
        coveredCases: [`'"uni.close" | transloco'`, `'"uni.close" | transloco : variable'`],
      },
    ],
    typescript: [
      {
        formula: (key: string): string => `_\\(\\s*${key}\\s*\\)`,
        type: 'both',
        coveredCases: [`_('uni.close')`, `_("uni.close")`],
      },
      {
        formula: (key: string): string => `transloco\\.translate\\(\\s*${key}\\s*(,\\s*.*\\s*)?\\)`,
        type: 'both',
        coveredCases: [
          `transloco.translate('uni.close')`,
          `transloco.translate("uni.close")`,
          `transloco.translate('uni.close', variable)`,
          `transloco.translate("uni.close", variable)`,
        ],
      },
    ],
  },
});

Deps

typescript 5
node 18

shaharkazaz commented 1 year ago

@Celtian Thanks for sharing

Celtian commented 11 months ago

I have already made some script. Probably this is not as powerful as this library, but I hope it helps as temporary solution.

/* eslint-disable @typescript-eslint/no-explicit-any */
import { readFileSync, writeFileSync } from 'fs-extra';
import { globStream } from 'glob';
import * as path from 'path';

interface AbstractUpdateLangConfig {
  encoding: BufferEncoding;
  defaultValue: string;
}

interface AbstractKeyStoreConfig {
  keyStore: Set<string>;
}

interface AbstractCwdConfig {
  cwd: string;
}

interface AbstractLangsConfig {
  langs: string | string[];
}

interface FindOccuranceConfig extends AbstractKeyStoreConfig {
  regex: RegExp[];
  fileContent: string;
}

interface UpdateLangConfig extends AbstractUpdateLangConfig, AbstractKeyStoreConfig {
  langPath: string;
}

interface UpdateLangsConfig
  extends AbstractUpdateLangConfig,
    AbstractKeyStoreConfig,
    AbstractCwdConfig,
    AbstractLangsConfig {}

interface ParserConfig {
  formula: (key: string) => string;
  type: 'single' | 'double' | 'both';
  coveredCases: string[];
}

interface MainConfig extends AbstractUpdateLangConfig, AbstractCwdConfig, AbstractLangsConfig {
  dryRun: boolean;
  source: string | string[];
  regex: {
    html: ParserConfig[];
    typescript: ParserConfig[];
  };
}

const flattenJson = (obj: any, parentKey: string = '', separator: string = '.'): Record<string, string> => {
  let result: Record<string, string> = {};

  for (const key in obj) {
    if (obj.hasOwnProperty(key)) {
      const newKey = parentKey ? `${parentKey}${separator}${key}` : key;

      if (typeof obj[key] === 'object' && obj[key] !== null) {
        const flattenedSubObj = flattenJson(obj[key] as any, newKey, separator);
        result = { ...result, ...flattenedSubObj };
      } else {
        result[newKey] = obj[key] as string;
      }
    }
  }

  return result;
};

const findOccurance = (config: FindOccuranceConfig): void => {
  for (const rx of config.regex) {
    let matchesMarker;
    while ((matchesMarker = rx.exec(config.fileContent)) !== null) {
      const key = matchesMarker.groups?.['content']?.trim();
      if (key) {
        config.keyStore.add(key);
      }
    }
  }
};

const updateLang = (config: UpdateLangConfig): void => {
  const langFileContent = readFileSync(config.langPath).toString();

  const flattenedLang = flattenJson(JSON.parse(langFileContent));

  const langKeyStore = new Map<string, string>();

  for (const [k, v] of Object.entries(flattenedLang)) {
    langKeyStore.set(k, v);
  }

  const result: Record<string, any> = {};

  for (const key of Array.from(config.keyStore).sort((a, b) => a.localeCompare(b, 'en'))) {
    const keys = key.split('.');
    let currentObj: Record<string, any> = result;

    for (let i = 0; i < keys.length; i++) {
      const currentKey = keys[i];
      currentObj = currentObj[currentKey] =
        currentObj[currentKey] || (i === keys.length - 1 ? langKeyStore.get(key) || config.defaultValue : {});
    }
  }

  writeFileSync(config.langPath, JSON.stringify(result, null, 2), {
    encoding: config.encoding,
  });
};

const updateLangs = (config: UpdateLangsConfig): void => {
  console.log('\n✔ Writing result into language files');

  const files: string[] = [];

  const langFilesStream = globStream(config.langs, {
    cwd: config.cwd,
  });

  langFilesStream.on('data', async (langPath) => {
    files.push(langPath);

    updateLang({
      defaultValue: config.defaultValue,
      encoding: config.encoding,
      keyStore: config.keyStore,
      langPath: path.join(config.cwd, langPath),
    });
  });

  langFilesStream.on('end', () => {
    console.log(`ℹ Keys were were updated in:\n`);

    console.table(files);

    console.log('\n🌵 Done! 🌵\n');
  });
};

const main = async (config: MainConfig): Promise<void> => {
  console.log('Starting Translation Files Build 👷🏗');

  console.log('\n✔ Extracting Template and Component Keys 🗝');

  const keyStore = new Set<string>();
  let filesCount = 0;

  const rxSingleQuotes = /'(?<content>([^'\s]|\\')+)'/;
  const rxDoubleQuotes = /"(?<content>([^"\s]|\\")+)"/;

  const filesStream = globStream(config.source, {
    cwd: config.cwd,
  });

  filesStream.on('data', async (filePath) => {
    filesCount++;
    const fileContent = readFileSync(path.join(config.cwd, filePath), {
      encoding: config.encoding,
    }).toString();

    const createRegex = (parser: ParserConfig): RegExp[] => {
      if (parser.type === 'single') {
        return [new RegExp(parser.formula(rxSingleQuotes.source), 'g')];
      } else if (parser.type === 'double') {
        return [new RegExp(parser.formula(rxDoubleQuotes.source), 'g')];
      }
      return [
        new RegExp(parser.formula(rxSingleQuotes.source), 'g'),
        new RegExp(parser.formula(rxDoubleQuotes.source), 'g'),
      ];
    };

    if (filePath.endsWith('.ts')) {
      findOccurance({
        regex: [...config.regex.typescript.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    } else if (filePath.endsWith('.html')) {
      findOccurance({
        regex: [...config.regex.html.map(createRegex).flat()],
        fileContent,
        keyStore,
      });
    }
  });

  filesStream.on('end', () => {
    console.log(`ℹ ${keyStore.size} keys were found in ${filesCount} files.`);

    if (!config.dryRun) {
      updateLangs({
        cwd: config.cwd,
        defaultValue: config.defaultValue,
        encoding: config.encoding,
        langs: config.langs,
        keyStore,
      });
    } else {
      console.log(`ℹ Dry run activated. Language files will not be updated.`);

      console.log('\n🌵 Done! 🌵\n');
    }
  });
};

main({
  dryRun: false,
  encoding: 'utf-8',
  defaultValue: '███',
  cwd: path.join(process.cwd(), '..'),
  source: ['projects/portal/src/app/**/*.ts', 'projects/portal/src/app/**/*.html'],
  langs: ['projects/portal/src/assets/i18n/*.json'],
  regex: {
    html: [
      {
        formula: (key: string): string => `{{\\s*${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?}}`,
        type: 'both',
        coveredCases: [
          `{{ 'uni.close' | transloco }}`,
          `{{ "uni.close" | transloco }}`,
          `{{ 'uni.close' | transloco: variable }}`,
          `{{ "uni.close" | transloco: variable }}`,
        ],
      },
      {
        formula: (key: string): string => `"${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?"`,
        type: 'single',
        coveredCases: [`"'uni.close' | transloco"`, `"'uni.close' | transloco : variable"`],
      },
      {
        formula: (key: string): string => `'${key}\\s*\\|\\s*transloco\\s*(:\\s*.*\\s*)?'`,
        type: 'double',
        coveredCases: [`'"uni.close" | transloco'`, `'"uni.close" | transloco : variable'`],
      },
    ],
    typescript: [
      {
        formula: (key: string): string => `_\\(\\s*${key}\\s*\\)`,
        type: 'both',
        coveredCases: [`_('uni.close')`, `_("uni.close")`],
      },
      {
        formula: (key: string): string => `transloco\\.translate\\(\\s*${key}\\s*(,\\s*.*\\s*)?\\)`,
        type: 'both',
        coveredCases: [
          `transloco.translate('uni.close')`,
          `transloco.translate("uni.close")`,
          `transloco.translate('uni.close', variable)`,
          `transloco.translate("uni.close", variable)`,
        ],
      },
    ],
  },
});

Deps

typescript 5
node 18

Script moved here: https://www.npmjs.com/package/ngx-i18n-extract-regex-cli

kekel87 commented 11 months ago

Since Angular 16 there are also Self-Closing-Tags components which do not work with this lib: https://github.com/ngneat/transloco-keys-manager/issues/155

shaharkazaz commented 10 months ago

@kekel87 Yes that's correct. This library needs to be migrated to ESM to support the new Angular compiler distribution.

The community is welcome to open a PR and make this change. people like to complain but not contribute. This library and Transloco consume a lot of time but don't even get sponsored. If I have the time I might get to it one day. In the meantime, if someone wants this feature, they are welcome to do the work.

kekel87 commented 10 months ago

@shaharkazaz I think I've succeeded, I've opened a PR. Let's chat inside, when you can. All the best

mackelito commented 10 months ago

any updates?

lkuendig commented 7 months ago

thanks for your work so far! transloco-keys-manager find seems to have problems to find the right missing keys too. seems this dependency needs also an update

shaharkazaz commented 7 months ago

Control flow should be supported in the latest versions. @austinw-fineart @lkuendig Is the original issue resolved? If there is an issue regarding find I think let's open a new detailed issue about it.

austinw-fineart commented 7 months ago

Yep, it looks to be working. Closing as fixed.

jsverse / transloco-keys-manager