RFC: New i18n implementation due to the app-router migration

[grzpab]

Is your proposal related to a problem?

When cal.com moves to Next.js' App Router API, i18n will no longer be "natively" supported by that API, namely .locale, .locales, .defaultLocale and .domainLocales properties of the router object will not be accessible.

Describe the solution you'd like

I thought of an architectural change that moves the resolving of translation functions from client-side components to server-side components.

This could be achieved by the following principles:

1. for any session, the language of the page will be decided based on:
   1. the language in the user settings, if the user is logged in; we can get that information using the session cookie,
   2. the accept-language header otherwise,
2. the language translations will be resolved only in the server-side code, based on the session language; the client-side code will be agnostic of the concept of translation.

The technical plan consists of the following steps:

1. the installation of the i18next module https://github.com/i18next/i18next, which contrary to the name, is agnostic of Next.js (the parent library of the next-i18next that Cal.com have been using),
2. loading of all supported languages during the booting of the Next.js server using https://www.i18next.com/overview/api#getfixedt, so all requests have access to these languages as global map of language/region codes mapped to the corresponding language objects (the exact way is to be decided, but the requirement is that this map is created once, is immutable and is available to every request),
3. every page uses a helper function that obtains the language/region pair from the request and matches it with the appropriate language object in the global language map. The selection is based on:
   1. the user language settings inferred from the session cookie in the request,
   2. the header called accept-language,
4. the language is then accessible to every server-side rendered component in a particular page,
5. the react-i18next, next-i18next libraries and associated middlewares and hooks will have to be cleaned up, each t and Trans usage will have to be changed into a usage of the language object mentioned beforehand (this will be codemodable):
   1. around 360 usages of t from useLocale from @calcom/lib/hooks/useLocale (this is the Cal.com hook),
   2. around 30 usages of <Trans> from next-i18next ,
   3. around 10 usages of t from useTranslation from react-i18next (in AppConfiguration).
6. trivia: there’s probably a significant number (possibly 1000+) of unused translations, e.g. organization_banner_title (removal can be codemoded as well).

Describe alternatives you've considered

In theory, the translation could stay in the client-side components, but it has the following drawbacks:

1. The next-i18next library will have to be replaced anyway, as outlined here: https://github.com/i18next/next-i18next/discussions/1993#discussioncomment-4337519
2. Currently, in order to display any translations on Cal.com, the language/region pair has to be recognized (based on the session cookie or the accept-language header), the JavaScript code has to fetch the correct translation JSON from the CDN, parse it, and make React display the translations by rerendering of components. Interestingly enough, if the accept-language header doesn't match the user settings' language, the app first fetches the former language translation pack and then the latter. The part with requests to the CDN, parsing JSON and rerendering of components (which costs time) will not occur when translating on the server-side.
3. If the translation packs are placed on the CDN, it definitely incurs some traffic costs and requires proper CDN asset managements (e.g. when a new translations are needed, all CDN entries need to be invalidated; what should the app do if a particular translation phrase is not available yet because the translation pack has not been refreshed?). With server-side rendering, Cal.com doesn't need to care about asset management and missing translation phrases.
4. Some components could be cached on the server-side even if they rely on translation (here I am thinking about React.memo) but I have yet to prove that such caching would yield noticeable benefits.

Additional context

N/A

Requirement/Document

N/A

[github-actions[bot]]

Thank you for opening your first issue, one of our team members will review it as soon as it possible. ❤️🎉

[zomars]

Sounds like a solid plan. I was wondering what are the main differences in this approach VS the i18next blog post approach?

AFAIK we're not using the /[locale]/ prefix right?

[grzpab]

Hello, we took a deeper dive and came up with two solutions superseding the original one. At the core of each are two principles:

1. not removing the next-i18next and associated libraries
2. calculating as much on the server instead of the client.

Both will work using the Pages or the App Router. Both should reduce the loading time of a page for every user. It will be faster to see the results from Solution A.

Facts:

1. Calcom uses a JWT (JWE) over cookies that contains session information including the user language.
2. Any information stored in the JWE can be extracted server-side instantaneously by having the symmetric secret AES key it was created with, without any database call.
3. The business requirement is that we don’t pass language and region as the dynamic segments in the URL
4. We cannot access headers or cookies in the getStaticProps function (of each page)
5. We can access header and cookies in the getServerSideProps function (of each page)
6. We can access the dynamic segments (params) in both getStaticProps and getServerSideProps functions
7. To use Static Site Generation (SSG) we need to use getStaticProps but we cannot use getServerSideProps
8. We cannot access request and response in getInitialProps of the _document server-side component
9. Based on Facts 3, 4, 5, 6, 7 we cannot use SSG if the we rely on information passed in headers and cookies

Solution A (Translations sourced from the CDN)

Steps:

1. Instead of fetching the session information by calling an endpoint by the client code, the session information can be retrieved on the server by inspecting the JWT (by Facts 1 and 2):
   1. this saves time ~300ms currently needed for the fetching, which is executed ~500ms after the page has started loading
   2. we can pass session information to the pages using getServerSideProps function
   3. we don’t prefetch the session using the TRPC ssgInit functionality
2. When a user requests the page, we calculate the locale in getServerSideProps instantly on the server-side using:
   1. the JWT token (only for the logged-in users)
   2. the accept-language header
3. Having obtained the language, we emit the following tag in the head part of the HTML: <link rel="preload" fetchpriority="high" as="fetch" crossorigin="anonymous" href="..." />with the href pointing to the proper translation pack based on the language & calcom api version
   1. the browser will preload and cache the resource automatically when it sees the tag in the HTML,
   2. we won’t need to wait for the client-side JS to load and execute and fetch the resource which happens at the earliest ~500ms after the page has started loading.
4. Based on the session information obtained in Step 1b, the client-side call will implicitly fetch the (same) resource as in Step 3, using the cache: "default" parameter:
   1. if the resource has been loaded and cached before, then it will be available instantaneously
   2. else, its loading should have been started by the browser in Step 3.
5. Once the translation pack has been loaded, we change the global state of the translation library next-i18next and allow React to refresh all the components that contain translated phrases.

Solution B (Translations Sourced from the Server)

Steps:

1. Step 1 is the same as in Solution A
2. Step 2 is the same as in Solution B
3. Within each page (within the getServerSideProps) we retrieve the translation pack for the obtained language and the proper namespace correlated with the page, meaning each page will have its own namespace and some translation phrases will repeat between different pages. This translation pack is the fed into the server page component.
   1. we considered another solution with loading all the translation packs in a middleware (that will act as a global store for translations on the server) but this has two problems:
      1. the middleware are calculated in the Edge Runtime that has no access to the file systems (from which the next-i18n library loads them) so we would have to compile the JSON files within the app bundle and serve them from the compiled code
      2. all the translations are huge, will probably take around ~10MB of RAM to serve them
4. We can use the translations in the server-side and the client-side components as they will be already resolved in at the server-side, you can see the example here:
   1. the Pages Router: https://github.com/grzpab/i18n-greg-test/commit/9c7bfafed6a49d4ac302a59ea531d7dcaf589514
   2. the App Router: https://github.com/DmytroHryshyn/dive-in-next-13-tests/commit/9c7b7411e835f443a305b2d68c287af5d9bc2ce0

Perhaps we should discuss if either solution is acceptable to you.

[zomars]

Awesome write up @grzpab Thanks for the input. IMO Solution A would be the least effort maximum impact don't you think?