amannn / next-intl

🌐 Internationalization (i18n) for Next.js
https://next-intl-docs.vercel.app
MIT License
2.09k stars 195 forks source link

Automatic tree-shaking of messages #1

Open amannn opened 3 years ago

amannn commented 3 years ago

Messages can either be consumed in Server- or Client Components.

The former is recommended in the docs, as it has both performance benefits as well as a simpler model in general since there is only a single environment where code executes. However, for highly interactive apps, using messages on the client side is totally fine too. The situation gets a bit hairy when you need to split messages based on usage in Client Components (e.g. by page, or a mix of RSC / RCC).

Ideally, we'd provide more help with handing off translations to Client Components.

Ideas:

  1. User-defined namespaces
  2. Manually composed namespaces
  3. Compiler-based approach

(1) User-defined namespaces

This is already possible (and shown in the docs), but unfortunately not so maintainable. For single components it can be fine however, so when you have many small interactive leaf components, this is quite ok.

(2) Manually composed namespaces

This worked well for the Pages Router and was documented, but unfortunately doesn't work with RSC.

Apollo has a similar situation with static fragments: https://github.com/apollographql/apollo-client-nextjs/issues/27

A potential solution could be to use a build-time macro that is evaluated separately.

(3) Compiler-based approach

See https://github.com/amannn/next-intl/issues/1#issuecomment-1497645889. Similar to fragment hoisting in Relay, if the collection of namespaces happens at build time, we could provide this list to Server Components. This seems to be the best option.

Current progress:

See also: Dependency tree crawling

amannn commented 3 years ago

An alternative (or possibly addition) is a concept based on the recently announced server components.

Code like this:

function LatestFollower({user}) {
  const t = useTranslations('LatestFollower');

  function onFollowClick() {
    UserAPI.follow(user.id);
  }

  return (
    <>
      <Text>{t('latestFollower', {username: user.name})}</Text>
      <IconButton
        aria-label={t('followBack')}
        icon={<FollowIcon />}
        onClick={onFollowClick}
      />
    </>
  );
}

… could be turned into:

function LatestFollowerServer({user}) {
  const t = useTranslations('LatestFollower');

  return (
    <LatestFollowerClient
      $1={<Text>{t('latestFollower', {username: user.name})}</Text>}
      $2={t('followBack')}
      $3={<FollowIcon />}
    />
  );
}

function LatestFollowerClient({$1, $2, $3, user}) {
  function onFollowClick() {
    UserAPI.follow(user.id);
  }

  return (
    <>
      {$1}
      <IconButton aria-label={$2} icon={$3} onClick={onFollowClick} />
    </>
  );
}

These are the interactive features that React currently has:

Most are easy to detect, but event listeners seem a bit tough, since it could potentially be a render prop. A convention like on[A-Z].* would definitely help here.

amannn commented 3 years ago

For very performance critical apps, it might make sense to use components that make use of next-intl only on the server side and to provide labels as strings to interactive client components. That way you wouldn't have to tree shake anything.

amannn commented 2 years ago

A manual approach seems easy and reliable: https://github.com/amannn/next-intl/pull/62

nbouvrette commented 2 years ago

Thanks @amannn for sharing this with me. For those who might be interested, we built a similar Babel plugin solution that works on Next.js: https://github.com/Avansai/next-multilingual

Of course, as called out, one of the drawbacks is that all locales are loaded in the same bundle which increases the size. I am planning to try to find a solution as we are adding an SWC version of the plugin.

amannn commented 1 year ago

@yashvesikar just shared a POC for a webpack plugin that extracts messages based on page entry points with me: https://github.com/yashvesikar/next-intl-translation-bundler-example

I'm still mind blown 🤯

AFAICT what we'd have to do (assuming we'd implement this for the app directory) is:

  1. Detect all entry points. In the POC, every file is treated as an "entry point", meaning that it will emit translations. This would need to be adapted to only handle modules that have the 'use client' banner as these are the boundaries where we're switching to the client side.
  2. For every entry point, traverse the module graph and find all references to useTranslations/t and collect the namespaces.
  3. Emit the translations per entry point.

The other part is that this is a webpack plugin, no idea if these will be supported by Turbopack or if we'd have to implement it in Rust instead (if Turbopack even supports plugins at this point).

Edit: An improved version of the webpack plugin that supports the App Router was implemented in https://github.com/jantimon/next-poc-i18n-split.


Another related project: https://lingui.dev/guides/message-extraction#dependency-tree-crawling-experimental

yashvesikar commented 1 year ago

@amannn Unfortunately, I don't have much knowledge of the app directory and all that RSC entails. I can speak to some of the advancements I implemented to this logic in the past and some of the ones I didn't get to.

// src/components/my-component/i18n/index.ts // loader can check that the keys match the ones in the main locale json let translations = {title: "title"} as const

// src/components/my-component/my-component.tsx import { translations } from "/.i18n

export const MyComponent() => { // by modifying the useTranslations hook we can make t(key) have key be typeof translations const t = useTranslations(translations);

return(
    <div>{t(translations.title)}</div>
)

}



- Figuring out how to get Webpack to load the translation files as dependencies. 
  Right now in the POC, I made the server is loading the files from the disk using `fs.readFileSync`. This is probably the worst way to read that data for high-traffic scenarios because it is a blocking call. Even making that call async will be much better, even better _might_ be to figure out how to have Webpack resolve the dependency itself such that the translations for a given page/path are available to the `getServerSideProps` function the same way any other import would be. I spent a little time looking into that but Webpack beat me 😅
  An open question is whether caching would be better done this way or via a separate network request with a long max-age.

- Improving the HMR, right now the HMR in dev _kinda_ works. The POC plugin will build the right translation output files the first time the page is loaded in dev, but if the translation JSON file is changed the server needs to be restarted. 

Some known limitations are that the Webpack parser is very limited and fickle. It's not a great way to traverse the AST to get all the keys, but since it's a single toolchain it is nice. I don't have any experience in SWC or TurboPack parsers but I have a hard time imagining they are worse than Webpack.

There are a few more things but I will leave it here for now. Sorry for the wall of text, happy that this may be useful to the project :) 
amannn commented 1 year ago

Hey @yashvesikar, thanks a lot for chiming in here and sharing your thoughts in detail!

Support for colocated translations by component

That's an interesting one, I think @nbouvrette is exploring something similar to this.

At least for next-intl, I'm currently not pursuing this direction though and rather try to make it easy to work with message bundles per locale as from my experience this integrates well with using translation management systems. We do have type checking though btw.!

Improving the HMR

Good point! That's something that's working fairly well currently, I'd have to investigate this for sure.

I don't have any experience in SWC or TurboPack parsers but I have a hard time imagining they are worse than Webpack.

😆 Well, I've read that Turbopack is adding support for Webpack loaders, I'm not sure what the story for plugins is though. This topic is generally a bit futuristic at this point and Turbopack is still quite experimental from what I know. At least for the time being, a webpack plugin would definitely be the way to go.

(continuing from our Twitter thread) in nextjs every page is an entrypoint + a main entrypoint for shared assets. so extracting all keys for a page is the same as by entrypoint. the webpack docs are simultaneously really useful and not useful enough 😅

Hmm, I just console.log'd the entry points—and I can confirm the part with the pages being entry points! But e.g. if pages/A.tsx imports components/Profile.tsx, how can I find this reference to process the dependency graph recursively?

Generally I'm currently still quite busy with the RSC integration for next-intl, but your thoughts are really helpful here, I hope to find some time to revisit this topic in the future! Made me laugh the other day when I saw that this was issue #1! 😄

nbouvrette commented 1 year ago

Great to see I'm not the only one wanting to solve this problem. I don't know if you guys saw but I split the next-multilingual logic that dynamically loads message modules into this package: https://github.com/Avansai/messages-modules

It uses Babel to dynamically inject the message in their local scope which is pretty neat but one optimization I would like to fix is that it loads all locales and not just the one being viewed... which of course would end up causing performance issues in the long run. But I think it's still better than a huge message file containing messages that are not even used

The big problem ahead is that from what I understood the app directory cannot use Babel... so basically I will need to rewrite this package in Rust so that it can work with SWC.

If you guys are interested to team up, I would be more than happy to get some help. Otherwise, I will most likely work on this later this year (Q3?) which should also give extra time to make the app directory more stable

gurkerl83 commented 2 months ago

Maybe something like like this can help to omit loading all translation files at ones, esp. the caching strategy is interesting here one variant of the server part. https://github.com/QuiiBz/next-international/blob/v2/packages/next-international/src/app/server/create-get-scoped-i18n.ts The client part especially how translations are instructed to load is also interesting, also this requires no plugin.

Maybe this helps Thx!

amannn commented 2 months ago

In next-international, getScopedI18n is about being able to provide a namespace, which is not supported by getI18n as far as I understand. Loading of messages upfront via createI18nServer/createI18nClient is the same for either API. That's at least my understanding.

next-intl handles both cases with one API via getTranslations(namespace?) and useTranslations(namespace?).

The current plan for this feature is to collect namespaces and keys with a compiler!

gurkerl83 commented 2 months ago

I apologize for the misunderstanding in my earlier message about scoped translations - I should have read the documentation more thoroughly. Scoped translations help streamline code by reducing redundancy.

getI18n vs getScopedI18n

const t = getI18n();
console.log(t('pages.settings.published'));
console.log(t('pages.settings.title'));

const t = useScopedI18n('pages.settings');
console.log(t('published'))
console.log(t('title'));

About the last sentence...

The current plan for this feature is to collect namespaces and keys with a compiler!

In which context, the issue about tree shaking or how next-intl currently handles internals?

Thx!

amannn commented 2 months ago

No worries! Yep, I agree that scoping translations is helpful. next-intl is heavily based on that too! 🙂

stylehuan commented 3 days ago

Is there any progress?