Per-locale bundles - Githubissues

tricoder42 commented 5 years ago

Discussion for RFC 003: https://lingui.js.org/rfc/003_per_locale_bundles.html

Introduction from RFC

Per-locale bundles are another build time optimization to reduce size of internationalized JavaScript applications.

Consider this example - we have an internationalized app which loads translation from external file:

// src/Notifications.js
import * as React from "react"
import { setupI18n } from "@lingui/core"
import { I18nProvider } from "@lingui/react"
import { Trans, Plural, date } from "@lingui/macro"

const i18n = setupI18n()
i18n.load("cs", import("./locale/cs/messages.json"))
i18n.activate("cs")

const Notifications = ({ now, count }) => (
   <I18nProvider i18n={i18n}>
      <h1><Trans>Notifications</Trans></h1>

      <p><Trans>Today is {date(now)}</Trans></p>

      <p>
         <Plural
            value={count}
            one={<>You have <strong>#</strong> unread notification</>}
            other={<>You have <strong>#</strong> unread notifications</>}
         />
      </p>
   </I18nProvider>
)

If we generate the production bundle in Engish locale, it will roughly look like this - components are removed and formatting components ( and date()) are replaced with runtime versions:

// build/Notifications.en.js
import * as React from "react"
import { Plural, date } from "@lingui/react"

const Notifications = ({ now, count }) => (
   <div>
      <h1>Notifications</h1>

      <p>Today is {date(now)}</p>

      <p>
         <Plural
            value={count}
            one={<>You have <strong>#</strong> unread notification</>}
            other={<>You have <strong>#</strong> unread notifications</>}
         />
      </p>
   </div>
)

So far the code looks very similar to the original one except the loading of message catalogs is removed completely.

Let’s take a look on other than source locale, for example Czech. The message catalog might look similar to this:

msgid "Notifications"
msgstr "Upozornění"

msgid "Today is {now, date}"
msgstr "Dnes je {now, date}"

msgid ""
"{count, plural, "
"one {You have <0>#</0> unread notification} "
"other {You have <0>#</0> unread notification}}"
msgstr ""
"{count, plural, "
"one {Máte <0>#</0> nepřečtenou zprávu} "
"few {Máte <0>#</0> nepřečtené zprávy} "
"other {Máte <0>#</0> nepřečtených zpráv}}"

If we generate the production bundle for Czech locale, it will look roughly like this - translations are applied at build time. Also, has all locale specific plural rules:

// build/Notifications.cs.js
import * as React from "react"
import { Plural, date } from "@lingui/react"

const Notifications = ({ now, count }) => (
   <div>
      <h1>Upozornění</h1>

      <p>Dnes je {date(now)}</p>

      <p>
         <Plural
            value={count}
            one={<>Máte <strong>#</strong> nepřečtenou zprávu</>}
            few={<>Máte <strong>#</strong> nepřečtené zprávy</>}
            other={<>Máte <strong>#</strong> nepřečtené zprávy</>}
         />
      </p>
   </div>
)

Per-locale bundles has zero footprint of internatinalization library - the code looks exactly the same have it would look like when no internationalization was used at all. The remaining runtime layer are utilities for formatting like plurals, dates and number formatting. There’s also no extra request to fetch locale files and no runtime parsing.

theKashey commented 5 years ago

Combinatoric explosion - per language bundles * per browser builds == 🥳

Plus it’s not quite clear how to detect the “locale” to use. For example for us it could be or browser setting, or user setting, we have to read from a server side first - that makes things(without SSR) more complicated.

tricoder42 commented 5 years ago

Combinatoric explosion - per language bundles * per browser builds == 🥳

What are per-browser builds?

Plus it’s not quite clear how to detect the “locale” to use. For example for us it could be or browser setting, or user setting, we have to read from a server side first - that makes things(without SSR) more complicated.

Don't assume it will fit to any scenario. It should be optional as described in RFC.

I expect there's gonna be global getLocale function, which will be injected by webpack plugin to the top of index.html. In such function, you can check localStorage, cookie for cached locale settings or call API, on the first load. That will make the first experience a bit slower, unfortunatelly. However, if you care about super fast load times, you're most probaly doing SSR.

danielkcz commented 5 years ago

Well, as much as great it sounds on paper, I cannot imagine it in practice. What about code splitting? A single bundle is easy, but I cannot imagine how to would instruct which chunk in which locale to load.

We are currently using code splitting for catalogs as well so only the catalog for a selected language is loaded. I am not exactly convinced it's a bottleneck to read a message from a compiled catalog to be used in render. It's just a plain object. Sure, some hashing mechanism could, in theory, improve lookup in a bigger object, but that would require some benchmark.

In my opinion, it's not worth it to do per-locale bundles.

async function loadCatalog(lang: string) {
  let catalog: Catalog
  if (process.env.NODE_ENV !== 'production') {
    // prettier-ignore
    catalog = await import(
      /* webpackMode: "lazy", webpackChunkName: "i18n-[index]" */
      `@lingui/loader!./locale/${lang}/messages.po`
    )
  } else {
    // prettier-ignore
    catalog = await import(
      /* webpackMode: "lazy", webpackChunkName: "i18n-[index]" */
      `./locale/${lang}/messages.js`
    )
  }
  i18n.load({ [lang]: catalog })
  logLang('loaded catalog %s', lang)
}

danielkcz commented 5 years ago

Oh and btw, what about service workers and caching for offline apps? By default, it caches all bundles. That would be quite an overkill to multiply the number of code bundles by a number of locales. That could lead to hundreds of MB of data downloaded (on the background). That's very sad for any metered connection.

tricoder42 commented 5 years ago

@FredyC Code splitting with per-locale bundles works out of the box. That's the beauty of it.

Forget about loading catalogs or splitting catalogs. There're no runtime catalogs in per-locale bundles.

To make it easier to understand: Imagine you have all source in src/ and you don't use internationalization at all. Now you copy src to src_cs/ and you hardcode translations into the code. You build the english version from src/ and the Czech version from src_cs/. Code splitting works as usuall, because there's nothing specific to i18n.

This approach works exactly the same, just the hardcode translations part is done under the hood in macro and you have a single tree of source code.

tricoder42 commented 5 years ago

@FredyC Btw, at the conference, Tom Occhino, the engineering director of the React group, mentioned that they're doing the same at Facebook. If you open dev console in facebook you can see per-locale chunks being loaded (notice the en_GB part):

Snímek obrazovky 2019-05-21 v 10 02 24

So, it's definitely possible, it definitely does work at scale. Now the questions is how to make it work in different environments (browser, SSR) and figure out all the details, like service workers (I have no idea how they work).

theKashey commented 5 years ago

Another moment - would it also increase built times. Significant.

tricoder42 commented 5 years ago

Obviously. Build time * number of locales. On 21 May 2019, 11:18 +0200, Anton Korzunov notifications@github.com, wrote:

Another moment - would it also increase built times. Significant. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

danielkcz commented 5 years ago

I do understand the underlying principle of per-locale bundles, but how would you inform bundler into which files to write those bundles? It might require bundler specific plugins, or not?

When the user changes language, I would rather download a small file with translations than a whole app code because a couple of strings has changed.

Honestly, I wouldn't most likely use it. One thing is Facebook which obviously does need to scale big time and they have good enough infrastructure to cover it. But does the regular apps need it? There is certainly a lot of ways to optimize app code and I think it usually helps much more. Per-locale bundles almost feel like micro-optimization in overall.

I think that SSR alone kinda beats the purpose of it alone as it renders markups with translations in place already.

I don't know that much about service workers either, I mostly used Workbox which is bundled with CRA and it indeed does download and cache all emitted bundles. It would be probably possible to configure differently, but it might require ejecting and as such it might be a fairly advanced feature for chosen ones :)

tricoder42 commented 5 years ago

I do understand the underlying principle of per-locale bundles, but how would you inform bundler into which files to write those bundles? It might require bundler specific plugins, or not?

The simplest solution is running the bundler once per each locale with different env var: env LINGUI_LOCALE=cs yarn build. Macro would replace strings with translations and webpack would generate chunks [locale]/[chunkName].js. From here there's lot of way to simplify it with custom plugins, but the main idea is the same.

When the user changes language, I would rather download a small file with translations than a whole app code because a couple of strings has changed.

That's what I would call "micro-optimization". How often do you switch the locale on a website? I usually just once - when I visit the website and it's not in my preferred locale. You could pick the preferred locale before rendering from header. Once the user switches the locale, you save it to cookie or local storage. That's one time cost.

But does the regular apps need it? There is certainly a lot of ways to optimize app code and I think it usually helps much more. Per-locale bundles almost feel like micro-optimization in overall.

Pick an app, any app with i18n, mesure the size and performance. Then remove internationalization completely. Remove message parser, remove code for loading messages, remove external catalogs completely and hardcode translations. Measure the size and performance again. The difference in size is the cost of internationalization. Everybody talks about it so I assume it's a problem even for smaller apps.

The main selling point is that you get an app translated into many languages, but the final size (from user point of view) is like the app wasn't localized at all. There's zero footprint. (To be precise, the only extra runtime code is detection of preferred language). Everything else is resolved at build time.

AFAIK the message catalogs can grow into hundred of kilobytes. If you want to avoid that, you need to figure how to split them automatically and then how to load them. It can be solved and it might be useful under some circumstances, it's just a different problem which still results in sub-optimal performance.

I think that SSR alone kinda beats the purpose of it alone as it renders markups with translations in place already.

SSR is only about initial render. When you start navigating, you're rendering the app in the browser and you also need to download additional chunks. There's lot of room for improvement.

I don't know that much about service workers either, I mostly used Workbox which is bundled with CRA and it indeed does download and cache all emitted bundles. It would be probably possible to configure differently, but it might require ejecting and as such it might be a fairly advanced feature for chosen ones :)

On the other hand you don't need to worry about refreshing the whole page ;) AFAIK service worker in CRA is in separate file, so you can customize it, but I have no knowledge about it or how to do that.

I don't expect solving all problems in first iteration. I just want to figure out API which works both for traditional runtime localization and build time localization so I don't have to publish another major release after next 2-3 months.

danielkcz commented 5 years ago

webpack would generate chunks [locale]/[chunkName].js

How? :) You need to somehow modify webpack config to do that. Even to add a plugin it's either about ejecting or using some of those override tools. The worst thing is if you forget or mess up this manual configuration part, it will basically write everything into a single bundle and you will get only the last locale in a row.

That's what I would call "micro-optimization". How often do you switch the locale on a website? I usually just once - when I visit the website and it's not in my preferred locale.

Fair enough, but it's still an unnecessary cost the user on a metered connection has to pay. Besides, right now I am able to switch language without a reload and it happens in a matter of milliseconds. You can hardly achieve such fluent experience when downloading the whole app code again, it would need some progress loader and whatnot, but it will always be slower.

How does it even work to replace existing memory code with a new one? There is no hot loader to do that.

AFAIK the message catalogs can grow into hundred of kilobytes

I guess it depends, we do have somewhat medium-to-large application and the biggest compiled catalog chunk has about 40kB. I don't think it will grow much more. Such size is something I wouldn't even bother to think about considering that there is 4.3MB in bundled app code (leaving out vendor libs). Cannot imagine that the user would be forced to redownload the whole thing, especially on mobile.

AFAIK service worker in CRA is in separate file, so you can customize it, but I have no knowledge about it or how to do that.

That's only a part that takes care of updating, but the whole Workbox setup is entangled into a build process (and Webpack) so it's not that trivial. I dare to say that without proper instructions and tools most people won't be able to configure it properly.

Another moment - would it also increase built times. Significant.

It's true, but it's fairly easy to build a single language for development purposes. The production builds are better handled in CI/CD anyway.

Either way, I hope it will be an optional feature :)

tricoder42 commented 5 years ago

How? :) You need to somehow modify webpack config to do that. Even to add a plugin it's either about ejecting or using some of those override tools.

Yes, you need to update webpack config. Decent frameworks allows you to do that, e.g. Next.js. You would need update webpack config even for automatic splitting of message catalogs as described in #503, so the configuration overhead is about the same.

Fair enough, but it's still an unnecessary cost the user on a metered connection has to pay.

You need to weight the unnecessary costs. Either one time cost when app is loaded with wrong locale or repeated cost, when you browse the website and you're loading extra localization data.

Besides, right now I am able to switch language without a reload and it happens in a matter of milliseconds. You can hardly achieve such fluent experience when downloading the whole app code again, it would need some progress loader and whatnot, but it will always be slower.

What experience are you talking about? Do you really switch locale so often that it matters? As a developer you might try it to see how cool it is, but how often users do it? Just once, when they visit the website and the locale guessed from headers is wrong.

How does it even work to replace existing memory code with a new one? There is no hot loader to do that.

I don't know what you mean. There's no reloading, you need to refresh the page to activate new locale.

I guess it depends, we do have somewhat medium-to-large application and the biggest compiled catalog chunk has about 40kB. I don't think it will grow much more. Such size is something I wouldn't even bother to think about considering that there is 4.3MB in bundled app code (leaving out vendor libs). Cannot imagine that the user would be forced to redownload the whole thing, especially on mobile.

Well, if your initial page load is 4.3MB, then you're right, you have completely different problem :)

I don't know the exact limit, but I think the initial page load should be below 200kb. So, the page is loaded, user pays 200kb and if the detection fails (i.e. the user uses different system locale than what's their native language), then they switch the locale and pay another 200kb. From here they start gradually loading the rest of the app.

I still believe that if you pick the locale from Accept-Language, you'll get the locale mostly right. Would be great to back it by real stats though.

Either way, I hope it will be an optional feature :)

The RFC includes examples of both builds, so it's definitely gonna be optional.

danielkcz commented 5 years ago

when you browse the website and you're loading extra localization data.

Care to elaborate? I load a single language bundle on initial app load. If the user changes language, I will load another. Nothing extra is ever getting loaded.

Well, if your initial page load is 4.3MB, then you're right, you have completely different problem :)

Sorry, that number is for a whole app. The initial one is 130kB. But as a user navigates throughout the app, other bundles are being loaded. It's true though that user would rarely change the language after clicking through the whole app :)

As a developer you might try it to see how cool it is

Yea, it's damn cool 😎 You are right, I am focusing on wrong things here :)

tricoder42 commented 5 years ago

Care to elaborate? I load a single language bundle on initial app load. If the user changes language, I will load another. Nothing extra is ever getting loaded.

per-locale bundle

function Component() {
   return <span>Ahoj <strong>světe</strong></span>
}

locale agnostic bundle

function Trans({ id, component }) {
   // extra code here
}

const messages = {
   "Hello <0>world</0>": "Ahoj <0>světe</0>"
}

function Component() {
   return <span><Trans id="Hello <0>world</0>" components={{0: <strong />}} /></span>
}

There're two parts of extra data:

constant overhead - localization library itself, ~3-5kb in case of LinguiJS
linear overhead - every message is roughly tripled. Once in source code, once as message key in catalog and once more as a translation. That may vary in length, but it has roughly the same length as message key. It could be optimized by minifying the IDs (#139), but there's still gonna be extra code compared to locale-specific build.

Now the question is what is worse.

theKashey commented 5 years ago

per locale builds with more than one locale in a build? Strings are quite fast to parse, and strings from a same language groups might compress better together.

So example we target ~20 languages. We might create bundles for 5-languages-in-bundle, ending with just 4 bundles, which would be not much bigger than single-language-bundle. (but this would require research about pairing and gziping language pairs)

danielkcz commented 5 years ago

Btw, how is it going to work server side? Right now we can build a static site and it can be hosted from anywhere. With per-locale bundles, what will decide which file to pull? Is it possible with some client-side code that is locale agnostic? I mean it would be annoying to download eg. "en" locale every time and then download "cs" when client-side code kicks in and figures out which one to use.

tricoder42 commented 5 years ago

Btw, how is it going to work server side? Right now we can build a static site

SSR and static sites are two completely different things.

SSR - there's no need for per-locale bundle, because you don't need to optimize for bundle size.

Static site - since the site is static, it's just HTML/CSS with locales already baked in. Static sites work on the same principle as per-locale bundles, they're just served on different urls.

btw that's why I want to have unified API which works with per-locale and locale agnostic bundles - you could generate server-side code, which loads locales dynamically to generate initial HTML and then it loads per-locale bundle.

danielkcz commented 5 years ago

Static site - since the site is static, it's just HTML/CSS with locales already baked in. Static sites work on the same principle as per-locale bundles, they're just served on different urls.

I did mean static with JS included that drives the logic, not things like Gatsby.

So you mean you have to go to site.com/cs or site.com/en? That would mean there needs to be some detection script sitting at "site.com" which will redirect the user to appropriate language version. That kinda slows things down slightly as it's unlikely people will be going to specific language versions directly.

Sorry, I am not that experienced in this field, never done it before, perhaps there is an easy path.

tricoder42 commented 5 years ago

So you mean you have to go to site.com/cs or site.com/en? That would mean there needs to be some detection script sitting at "site.com" which will redirect the user to appropriate language version. That kinda slows things down slightly as it's unlikely people will be going to specific language versions directly.

I was talking about static site, i.e. site without client side endering.

If you're talking about client side rendering, then you need to detect locale early. You can always cache it to cookie or localeStorage, if detection is time consuming (i.e. if it requires to call the backed). In most cases, guessing locale from ACCEPT_LANGUAGE header should be enough.

danielkcz commented 5 years ago

Yea, detecting (and persisting) is an easy part, but how do you load locale-specific bundles? Is it just going be the same construct I am using for a code splitting? https://github.com/lingui/js-lingui/issues/508#issuecomment-494280290

I mean sure, there is going to be refresh after changing the language, but some "bootstrap" script still needs to read the persistence and grab correct resources.

eliseumds commented 5 years ago

@FredyC a server-side-rendered app would detect the language from the Accept-Locale header and would render the script tags accordingly:

<script src="https://www.mycdn.com/en-AU/webpack-runtime.js"></script>
<script src="https://www.mycdn.com/en-AU/app.js"></script>
<script src="https://www.mycdn.com/en-AU/HomePage.js"></script>

All these files are aware of the language because they were built with an specific publicPath (https://www.mycdn.com/{LANGUAGE}/) so subsequent dynamic imports would just work. The idea is the same for a client-side-only app. You'd detect the language on the browser, persist it somewhere and then load the initial resources:

const detectedLanguage = '...';
const publicPath = `https://www.mycdn.com/${detectedLanguage}/`;
const scripts = [
  publicPath + 'webpack-runtime.js',
  publicPath + 'app.js'
];

scripts.forEach(loadScript);

Does that make sense?

The complexity here lies in the build process since it can become very slow. Ideally we wouldn't build CSS files for each language, for ex. That's why you wouldn't want to mess around with that if you're not coding a massive app that needs to be fast. Much worse if you're doing SSR.

nickluger commented 5 years ago

I would really like to see this feature implemented.

Some people (like me) just want to generate different bundles for different countries/domains at build/compile-time and not allow any language switching on a certain domain. The user has to visit a different domain to get the right language.

A well known website that does it is:

https://www.amazon.com/ https://www.amazon.de/ https://www.amazon.fr/ ...etc.

Many websites have event different features, deployment strategies, legal requirements, content and products, backends, even APIs etc. that all depend on the country you're operating in. (e.g. Amazon).

Changing just the "language" on buttons, controls and texts makes no sense in these cases. This especially applies to all websites, where user-generated content is predominant, so there's no translation anyway for the majority of the texts on the website.

Still, if 90% of the JS is the same on several sites, a common code-base makes sense. In this case, hard-coding the translated strings at compile time is (performance-wise) the best thing to do.

And the performance impact is quite severe:

As there's no catalogue-splitting yet, our compiled language catalogue is currently about 100KB and growing. This is pretty bad for mobile/2nd world connections.

The zero-footprint strategy proposed here by @tricoder42 makes a lot of sense here.

theKashey commented 5 years ago

All these files are aware of the language because they were built with an specific publicPath (https://www.mycdn.com/{LANGUAGE}/)

There are two languages - one is build-in (the one you have "used" in the code), and another one you may switch into.

The first one is "expected" to be on Application start, and the second one you may "await".

The second one

The second one is not a subject for the "real" code splitting - you may load all strings in a selected language without any "hurt" for the customer, as long as you will do it when the user changes settings in the interface setup - so some delay is "expected".

The first one

The problem is to create language chunks to match real "script" chunks. It's not easy, especially with different bundlers and different build setups. Let's go more stable way.

But the first one, the one used to initially render the page is very important, and should be compact and fast.

[prebuild] lingui extracts strings from files
[build] lingui prepends every file with LINGUI_ADD('#MODULENAME_STRINGS#')
[postbuild] lingui multiplicates the final bundle into "N" language dirrectories
[postbuild] lingui replaces every MODULENAME_STRINGS with actual strings for the module
we have got ideally optimized bundles

Drawbacks:

the best way to switching language is to switch bundles. However "the second one" could help here to keep user session active (not a problem for 99% customers)
modern/legacy bundles duplicate efforts and might break everything (IE11 is going to die in a year)

renchap commented 4 years ago

Running your bundler once per locale will generate the most optimized code, but will incur a big slowdown in your build process. An alternative, which should also be easy to implement, is to build it once using a placeholder catalog and then replace this placeholder with each locale's catalog, creating the final bundles.

This is definitely less optimized, but in practice if you manage a lot of languages you do not want your build to take dozen of minutes.

More details about this problem and how Etsy solved it: https://codeascraft.com/2020/02/03/production-webpack-builds/

tricoder42 commented 4 years ago

This is great article, thank you @renchap 👍 Right now I don't work on this feature, as I'm busy with #334. I'll definitely review it later.

theKashey commented 4 years ago

Thank you @renchap - I was looking for this article for a while. I could agree - the best code with translations, is code without translations, ie when they "disappear". As well as probably this solution is the only one truly bundler independent. However I believe that we could do better (working on it right now)

ScriptedAlchemy commented 4 years ago

Just a heads up, Module Federation will likely enable per-locale bundling capabilities. Youd have to use the lower-level API, but I think it should be able to do this

renchap commented 4 years ago

This is not directly related to this issue, but an idea I had to reduce the number of JS loaded for the user would be to have per-sourcefile locales. Code-splitting is now very common and you might have this (simplified) situation where half of your app is in one bundle and half of it is in another bundle. But your locales are all in one place, and the translations for your whole app needs to be loaded, even those for the bundles / chunks that are not currently loaded. Inlining the translation in locale-specific outputs would solve this, but I am wondering if there are other ways to do so.

What I am imagining is that each sourcefile "depends" on its translated strings, like with a virtual import : import translatedString from "lingui/locales/<string identifier>", and then your bundler can do its job and store the translations in the same chunks as the source code they are used into (or in common chunks if they are use in multiple places). This seems to be incompatible with the catalog concept, but I am wondering if people explored this path already.

Sorry for going off-topic in this issue, please tell me if this is worth opening a new issue to discuss this further, or if this is a dumb idea as well ;)

theKashey commented 4 years ago

There is another issue for it - https://github.com/lingui/js-lingui/issues/503, I even have 90% of code needed to do the stuff, and it's more or less based on "virtual files". I decided to make my life a bit more complex and support in-fly language change. As a result, I haven't finished the task in time.

Theoretically, this is a good task for module federation to make all heavy lifting for this as long as remotes are those "per-language" "virtual files". @ScriptedAlchemy 🤞

renchap commented 4 years ago

Ho I missed this issue, nice work then, cant wait to see how it ends up working 👍

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ScriptedAlchemy commented 4 years ago

Yeah this is actually perfect for module federation. You can literally write module router module gates and point even complex requirements to other modules in webpack.

For this case though, remotes with module federation would make this quite simple to perform. And translations could be shared between separate apps, if you were to deploy or maintain MFE or just multiple separate applications

tricoder42 commented 4 years ago

@ScriptedAlchemy I don't know much about module federation. Could you please create a PR with a proof of concept?

ScriptedAlchemy commented 4 years ago

I'd need calls with the authors. Someone who knows how the loader works or whatever / has an idea how to fit my design together. If someone is willing to partner.

We can do a zoom call maybe @theKashey we could do this?

I'm burning out so I've gotta cut back on OSS.

What I can say is: please persevere and use MF for this splitting. Anything else put forth is gonna be complex and way more work. MF was designed to do this stuff with a single import function.

We also have the ability to greatly expand lingui and its capabilities of MF powers this.

MF will likely just handle most of the hard part.

Who is dedicated and willing to partner, I'll help on MF.

Realistically it's probably an hour or 2 of work

tricoder42 commented 4 years ago

@ScriptedAlchemy Got it. When you're available for a call? I'm in UTC+1 timezone, but I could be available anytime. Feel free to reach me by email.

ScriptedAlchemy commented 4 years ago

I'm unavailable Monday. But will see if there's a slot that'll work here this week.

ScriptedAlchemy commented 4 years ago

Okay so a little more though for the meeting. If there's a branch somewhere in here that has some concept of splitting. That'll help.

We can create virtual modules.

I highly suggest reading this: https://medium.com/swlh/webpack-5-module-federation-a-game-changer-to-javascript-architecture-bcdd30e02669?source=friends_link&sk=c779636999c8644a7fc0df3aaa96223e

https://medium.com/dev-genius/module-federation-advanced-api-inwebpack-5-0-0-beta-17-71cd4d42e534?source=friends_link&sk=70658eb0bf58dfcc5ce534cb1cd78b1f

Id also recommend taking a look at my readme over here, there's some good info: https://github.com/module-federation/module-federation-examples

But I'm hoping there's a possible way to make MF do all the splitting.

In the MF plugin, I can basically module.export a SPA, or many, and have them work together as a monolith.

There are two implementation tactics I see right away.

1) we can use shared, to share a directory {name:"myapp",shared:"/translations/"} 2) we could expose them manually with exposes:{"./locale/en":thePath}, instead of shared

Herers what i configure as a top level plugin.

new ModuleFederationPlugin({
      name: "app1",
      filename: "remoteEntry.js",
      exposes: {
        //option one, treats it as a module.export
        "./translations/enh": "./src/translations/eng.json",
      },
      shared: {
        // option 2 a trailing slash will traverse the directory
        // shared distributes this as a package name at runtime. its like node_modules in the browser that many webpack apps can access, or a single monolithic webapck application can as well.
        translations: "./src/translations/"
      },
    }),

Now, here's what importing it would look like. This concept is known as the "host" using its own "remote".

If i expose it - the code that needs to be written into a file is

import('app1/translations/eng')

the second option

import('translations/eng')

It's important to note that you can use static imports as well the code thats share or exposes is always code-split regardless of async or static import

While federated code is async - you can still use static imports and require

import {thing} from 'translations/eng'

If static written imports will not work, for whatever reason - we have access to the low-level webpack API with is accessible in all runtime environments, just like MF

https://github.com/module-federation/module-federation-examples/blob/master/advanced-api/dynamic-remotes/app1/src/App.js

tricoder42 commented 4 years ago

@ScriptedAlchemy At the moment there isn't any branch with prototype of poc. I tried it a year ago but on completely new repository where I stripped down all unnecessary packages and kept only @lingui/loader. On top of that I played with Webpack plugins, but without useful result.

I guess if you'd just drive me through concepts of MF, I could figure out how to fit it in Lingui. To be honest, per-locale bundles aren't my top priority, but I'm hoping to use the same concept for splitting of compiled message catalogs.

Let's say we have a simple Next.js app with two pages:

locales
  - en.js  // all messages in the app
  - es.js
pages
  - index.ts
  - about.ts

The message catalog is imported like this:

import { i18n } from "@lingui/core"

function activate(locale) {
   const { messages } = await import(`../locales/${locale}.js`)
   i18n.load(locale, messages)
}

Right now, Next.js automatically splits code for each route, so you get something like this:

build
  - index.23748.js  // chunk name with some random hash
  - about.47893.js
  - en.83478.js
  - es.38747.js

I would like to figure out how to automatically split messages for each bundle:

build
  - index.23748.js  // app code which load translations from external file
  - index.23748.en.js
  - index.23748.es.js
  - about.47893.js
  - about.47893.en.js
  - about.47893.es.js

Even this would be huge improvement.

Next step is what's described in this issue — create a per-locale bundles:

build
  - index.23748.en.js // app code with english translations "baked" in, Lingui removed completely from runtime code
  - index.23748.es.js
  - about.47893.en.js
  - about.47893.es.js

I'm not sure how easy it would be to integrate it into Next.js or other frameworks.

@ScriptedAlchemy Would you mind if we reschedule the meeting to the next week? I see there's plenty of resources in https://github.com/module-federation/module-federation-examples (including a book), so meanwhile I'll do my homework, read what MF is and how to use it. Don't want to waste too much of your time what this is all about.

ScriptedAlchemy commented 4 years ago

As seen in the low-level api we can access the hosts own remote like window.app1.get('./translations/eng')

Whats not mentioned is that we can extend what app1.get actually resolves inside the webpack runtime.

We also have a concept for "virtual modules" build into module federation, though this implementation is not publically known well as its not really documented.

You could point webpack at a promise that you resolve at runtime with your own custom code from the host side, like this.

{
  remotes:{
    app1:  external: `promise new Promise(res => {

      const proxy = {get:(request)=> remote.get(request),init:(arg)=>{try {return remote.init(arg)} catch(e){console.log('remote container already initialized')}}}
      res(proxy)
      })`,
  }
}

At the same time, i could also write a remote that performed additional module routing logic when it attaches. the proxy concept would remain the same, however you can pretty much require different parts of an application with a module router replacing the get and handling the request.

This is a little dense, but it hopefully sheds some light on the level of flexibility we have in terms of how code can be attached and distributed at runtime.

ScriptedAlchemy commented 4 years ago

@tricoder42 yes we can do.

My examples only demonstrate very basic implementations. under the hood, i can make the system completely dynamic, far more than demonstrated.

NExt.js limits, we will have issues with next if we use the shared option without async importing it - if we use dynamic imports - it'll work on everything that webpack can compile.

Im on next.js, so id anticipate ensuring that it works with platforms im bound to haha Theres also this: https://github.com/module-federation/nextjs-mf to patch next for MF sharing.

Since the intent isn't to share react based components, we do not need 90% of that plugin :P

ScriptedAlchemy commented 4 years ago

Please reach out if you have any questions, Extensive work was done on the webpack core to enable Module Federation. We have other ways to access asynchronous externals if we needed to, though i usually use this for importing URLs like third party scripts on demand.

DM me on twitter if you need me/i don't respond,my github has a lot of traffic and noise at the moment

tricoder42 commented 4 years ago

@ScriptedAlchemy Nice! Perfect, thanks a lot. I'll read through the docs and examples in next few days 👍 Sounds exciting.

semoal commented 4 years ago

I would like to join you (to the meet) if it is not indiscretion, this feature will be super and a massive step-forward to improve localisation libraries

I'll start to read all tho documentation you provided Zack! Amazing

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

semoal commented 3 years ago

I up this, because this should be our focus to mid-end term. :)

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Bertg commented 2 years ago

I see this ticket got closed, but it seems it never got a proper solution. Should this be reopened?

lingui / js-lingui

Per-locale bundles #508

The second one

The first one