Urthen / case-sensitive-paths-webpack-plugin

Enforces case sensitive paths in Webpack requires.
MIT License
428 stars 45 forks source link

Avoid calling readdir() on same path multiple times #52

Open frantic1048 opened 4 years ago

frantic1048 commented 4 years ago

Recently I'm diagnosing build time on my project. I found case-sensitive-paths-webpack-plugin costs significant time (with speed-measure-webpack-plugin).

On a build with 4000+ modules.

CaseSensitivePathsPlugin took 3 mins, 47.68 secs
[CaseSensitivePathsPlugin] Total filesystem reads: 25316

After walking around CaseSensitivePathsPlugin's source, and extra log for cache hitting(at https://github.com/Urthen/case-sensitive-paths-webpack-plugin/blob/737dadd4b02e377296c9cd2d7c02703e09bb771a/index.js#L51-L53 ), I found below behavior(project path is redacted):

[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Reading directory /PROJECT_ROOT/node_modules/classnames
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/antd/es/_util
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/antd/es/tree
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/antd/es/upload
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/antd/es/upload
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/lodash
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/lodash
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/antd/es/_util
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/react-lifecycles-compat
[CaseSensitivePathsPlugin] Hit cache for directory /PROJECT_ROOT/node_modules/classnames

The path /PROJECT_ROOT/node_modules/classnames has been read for 20 more times before cache can be used, which makes cache did not reach its goal (reuse result for the same directory).

After made some subtle changes with cache-related logic, It now read each unique directory only once and gains significant speedup on my project(same machine, same project as above timing).

CaseSensitivePathsPlugin took 1 min, 58.28 secs [CaseSensitivePathsPlugin] Total filesystem reads(with visitedDir opt): 1487

Update 2021-02-23:

The subtle changes above(early changes on #53) are not correct, tests are failing. However, I'm trying to fix them.

frantic1048 commented 4 years ago

Changes to make pathsCache work more efficiently has been pushed to #53

I noticed #50 made similar changes to make cache work right. However, #53 only made pathsCache stores Promise object, and does not change the structure of code to solve this issue with small change as possible.