Broken Chunk-Splitting in Rollup

LorisSigrist commented 2 months ago

Context

Paraglide's current messages.js file cannot be properly chunk-split by rollup.

We assumed that the messages can be split / inlined automatically because they are exported separately. However rollup can only remove unused functions from modules, it cannot split functions from the same modules into separate chunks.

Because of this Paraglide currently doesn't properly split messages per-page.

Proposal

There are two things we could do:

Option 1

Implement chunk-splitting in Rollup (very large effort) https://github.com/rollup/rollup/issues/2512

Option 2

Generate a separate file for each message and re-export all of them from messages.js. This drastically increases the number of files in the compiled output but we wouldn't need to do any API changes.

|- messages.js.   //barrel file for messages/index/[messageId].js
|- messages/
  |- en.js        //barrel file for messages/en/[messageId].js
  |- en/
     |- messsage_one.js
     |- message_two.js
  |- index/
     |- message_one.js
     |- message_two.js

samuelstroschein commented 2 months ago

Is this a problem in turbopack and rolldown?

If not, we could just wait. Rolldown will eventually replace rollup in vite and turbopack replaces webpack.

Re option 2

can this be exposed as virtual modules in rollup/vite instead of the real fs to reduce "noise"?

LorisSigrist commented 2 months ago

Turbo / Webpack & EsBuild -> Works as expected
Rollup / Rolldown -> No Splitting

Rolldown is a reimplementation of Rollup so it inherits this missing feature. There is no telling how long it will take to add that. Until then all vite-based frameworks won't benefit from the Treeshakeing that should be enabled by paraglide. This fundamentally undermines our Pitch which is why I marked this as urgent.

Can this be exposed as virtual modules in rollup/vite

To do this we would need completely separate build outputs for rollup & non-rollup environments. This defeats the purpose of the FS based output IMO. If were to go with Option 2 I would prefer actually emitting the files.

samuelstroschein commented 2 months ago

Gut instinct is: fix this in rollup.

potentially 1000's of message files will slow down tools (linters etc) and degrade the DX of paraglide
degrading the DX for all paraglide users because a subset of users (rollup) has issues seems wrong
fixing this in rollup is beneficial to the entire rollup and web dev community

I would ask the maintainer of rollup (@lukastaegert) if he would fast-track a solution if you work on it, or we could also put a $xK bounty on the rollup issue.

Putting a substantial bounty on the rollup issue seems easiest.

@loris.sigrist what do you think about putting a substantial bounty on the issue in correspondance with @lukastaegert?

lukastaegert commented 2 months ago

To be honest I am sure I will not be able to divert the necessary resources in the next months independent of the bounty. We recently had a new high-quality contributor in Rollup, maybe I could interest @liuly0322 in the topic and the bounty? I would also be very interested to work on this but I am stretched to my limits. But I would provide any support I can give and feedback about potential approaches.

samuelstroschein commented 2 months ago

maybe I could interest @liuly0322 in the topic and the bounty?

That could work. @liuly0322 is solving https://github.com/rollup/rollup/issues/2512 interesting to you?

LorisSigrist commented 2 months ago

@loris.sigrist what do you think about putting a substantial bounty on the issue

Fixing this in Rollup itself would of course be the most elegant solution. However, speed is of the essence as well. While this issue persists Paraglide doesn't live up to it's promise.

We can fix this quickly with Option 2, any rollup update would need to come reasonably soon in order for it to be worth it for us.
If putting a bounty on this get's us that speed then I'm in favor, but if it's going to take months even with the bounty I would prefer going with Option 2.

liuly0322 commented 2 months ago

Hello, I am also new to rollup, but in my view, there is definitely a huge work for rollup to do to support chunk-splitting. Though esbuild has achived the goal,

The internal structure of esbuild and rollup is very different;
The bundle speed is also what rollup needs to take into consideration, as esbuild is written in native language, it has a speed advantage, and can do tree-shaking for every different code entry. However for rollup, maybe another approach is needed.

So though personally I think it would be great for rollup to support chunk-splitting, it may not be reasonably soon. I actually think a rollup/vite virtual module plugin is a good idea to balance performance and tree-shaking effect, feel free to ignore me as I don't know how many of your users are using vite :)

canseyran commented 2 months ago

From a user standpoint, who uses sveltekit + paraglidejs. As @samuelstroschein mentioned, I think implementing this feature in rollup is probably the most elegant solution and beneficial for the bundler in general. But the missing chunk splitting feature is already having a large impact on the amount of unnecessary code shipped to the user.

I would love to see the option 2, an experimental flag could be used to prevent breaking builds or causing performance issues, also taking into account that different build systems might handle barrel exports worse than others. Example flag: "--experimentalBarrelExports".

LorisSigrist commented 2 months ago

Paraglide-Astro and Paraglide-Sveltekit both use Paraglide-Vite under the hood. We could use that plugin to override imports from paraglide/messages.js and paraglide/messages/[lang].js with the barrel-file versions (virtually). The FS output would remain the same.

This would give us the Chunk-Splitting without needing to change the FS output. It would require some duplicate work on the compiler, but as long as it's a temporary fix it would be manageable IMO.

This is a hack, but it could work.
@samuel.stroschein what do you think? Is this what you meant when suggesting we use a rollup plugin?

samuelstroschein commented 2 months ago

@loris.sigrist nice. you found a way to do virtual modules in rollup e.g. not writing 1000s of files to disk in https://github.com/opral/monorepo/pull/2707 ?

LorisSigrist commented 2 months ago

Correct. FS files stay the same but chunk splitting works

opral / inlang-paraglide-js