search

embroider-build / ember-auto-import

Zero config import from npm packages
Other
360 stars 109 forks source link

Looking for some _hopefully very light_ maintainer support as I troubleshoot an annoyingly long warm boot #570

Closed Stasmo closed 1 year ago

Stasmo commented 1 year ago

Hello and thank you for reading my ticket. I don't want to bother y'all too much so I'll try to keep this ticket short and sweet. If it's missing some critical details please let me know. I am no expert in ember or babel or webpack. I am assuming I have misconfigured something.

Project details

I'm on 2.6.1 of the ember-auto-import package. Version 16.17 of node. Ubuntu running in WSL2. The project uses 20 in-repo engine addons that all use a common in-repo addon. Some of my in-repo engine addons are using addons from npm.

The problem

ember-auto-import-analyzer is run 66 times on warm boot, taking 207ms on average, about 14s total.

It is reading and parsing every file in my project, in-repo addons and npm addons on warm boot, every time. No changes made to any of the files or config in between warm boots. I stop the server and start it again immediately after and it's still reading and parsing every file.

My plan

I want to determine if it's possible to skip the analyzer for files that haven't changed since the last time the analyzer was run. I am looking through the source for this project and learning more about babel, but I wanted to reach out see if there's something obvious that I have missed, and if anybody has any pointers on where to get started.

Thank you!

Stasmo commented 1 year ago

OK I think this is a babel caching issue and probably not related to this project but still wondering if somebody has a hint for me.

ef4 commented 1 year ago

Try running with DEBUG=ember-auto-import:analyzer and see if it's printing "updating imports for..." vs "updating imports (the slower way) for...".

Stasmo commented 1 year ago

There are 2 plugins that don't support the fast analyzer.

After digging in a bit, it looks like the analyzer always starts with an empty FSTree, so it's always going to create a patch set that includes every file at start up. I think the only way around that is to skip the analyzer altogether at the babel level, or to introduce caching into the analyzer. I think I've just misconfigured babel somehow, because I can see it creating a new directory in my /tmp dir at startup and deleting it when the app stops instead of persisting it between starts.

Thanks for your time I'm going to close this ticket.

ef4 commented 1 year ago

When using the faster version (so any addons that use ember-auto-import greater than I think 2.2.0), the analysis happens in two phases. The expensive part, where we need access to the full parse, happens inside the existing babel pass and emits specially-formatted comments. This lets our state persist inside the babel cache.

Then the second pass grabs those specially formatted comments via a much cheaper search through the files that doesn't require a full javascript parse. This does need to look at every file at startup, because there's nowhere else we can cache the summary of what they contained that is guaranteed to be consistent with the babel cache.

I wonder how much file IO overhead is added by WSL. I know early versions of it were rough but I don't know how much it has improved. That may be part of the challenge here. Another is the large number of engines, which tends to exacerbate ember-cli's problem of duplicating work per consumer of each addon, instead of only scaling in the number of addons. There are ways to work around that, see for example cacheKeyForTree https://github.com/ember-cli/ember-cli/blob/c85fd3f4b252f964f707e60705a27f3ebda43526/lib/models/addon.js#L674-L690, which may be relevant in custom addons that are getting consumed many times, if those addons are doing anything custom in their own broccoli build.

Stasmo commented 1 year ago

Thanks Ed! There's an addon that's used in 12 of our engines that is being analyzed 12 times on boot. I'll look into the cacheKeyForTree config and see if I can get that deduped somehow.

Stasmo commented 1 year ago

Regarding WSL, there are some huge performance hits when working with mounted windows volume, but I am not leaving Linux-land for any of this so there shouldn't be any filesystem slowness. For example, running yarn install on my ember project on the Windows volume takes 30 minutes, and about 30 seconds within Ubuntu.