manifest plugins slows down build by 70 times

[mukeshsoni]

System Info

System: OS: macOS 13.6 CPU: (10) arm64 Apple M1 Pro Memory: 9.98 GB / 32.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 20.3.1 - ~/.nvm/versions/node/v20.3.1/bin/node Yarn: 1.22.17 - /opt/homebrew/bin/yarn npm: 9.6.7 - ~/.nvm/versions/node/v20.3.1/bin/npm Watchman: 2023.09.04.00 - /opt/homebrew/bin/watchman Browsers: Chrome: 119.0.6045.105 Safari: 17.0 npmPackages: @rspack/cli: ^0.3.10 => 0.3.10

Details

I am trying to use rspack-manifest-plugin as a replace for webpackmanifestplugin. Using rspack-manifest-plugin slows down our build from 5 seconds to 350 seconds. A whopping 70 times.

I will not be able to share a reproducible repo since the code is private. I can share some part of the rspack config file -

const path = require('path')
const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin')
const manifestPlugin = require('rspack-manifest-plugin').WebpackManifestPlugin
require('dotenv').config()

module.exports = function (env, argv) {
    const isProduction = argv.mode === 'production'
    var baseSrcPath = '.'
    const targetDir = '/path/to/target/directory'
    const sourceMapDir = 'sourcemaps'

    var plugins = [
        new manifestPlugin({
            writeToFileEmit: true,
        }),
        new ForkTsCheckerWebpackPlugin({
            typescript: {
                memoryLimit: 3096,
            },
        }),
    ]

    return {
        entry: {
            main: '/path/to/entry/file',
        },
        output: {
            pathinfo: isProduction,
            path: path.resolve(targetDir),
            publicPath: '/path/to/public/directory/',
            filename: '[name].[contenthash].js',
            sourceMapFilename: `${sourceMapDir}/[name].[contenthash].js.map`,
            chunkFilename: '[name].[contenthash].js',
        },
        resolve: {
            fallback: {
                net: false,
                fs: false,
                path: false,
                zlib: false,
                tls: false,
                http: false,
                https: false,
                url: false,
            },
           // Looks like this is the main culprit
            modules: [path.join(__dirname, '/src'), 'node_modules', 'src/pp/core/less/'],
            extensions: ['.js', '.jsx', '.ts', '.tsx', '.less'],
            alias: {
                // Some path aliases here
            },
        },
        mode: argv.mode || 'development',
        builtins: {
            presetEnv: {
                targets: ['Chrome >= 48'],
            },
        },
        module: {
            rules: [
                {
                    test: /\.mjs$/,
                    include: /node_modules/,
                    type: 'javascript/auto',
                    resolve: {
                        fullySpecified: false,
                    },
                },
                {
                    test: /\.less$/,
                    use: [
                        {
                            loader: 'style-loader',
                        },
                        {
                            loader: 'css-loader',
                            options: {
                                url: false,
                            },
                        },
                        {
                            loader: 'less-loader',
                            options: {
                                lessOptions: {
                                    url: false,
                                    javascriptEnabled: true,
                                    module: true,
                                },
                            },
                        },
                    ],
                },
                {
                    test: /\.(png|jpe?g|gif)$/i,
                    type: 'asset/resource',
                },
                {
                    test: /backbone\.js$/,
                    use: {
                        loader: 'imports-loader',
                        options: {
                            additionalCode: 'var define = false; /* Disable AMD for misbehaving libraries */',
                        },
                    },
                },
            ],
        },
        plugins,
    }
}

This is stopping us from moving from webpack to rspack. The manifest file generation is a must for us since we use that with our python backend to inject the right js bundle name into our html. The webpack build takes around ~37 seconds now. So rspack's build time of 5 seconds is a great win for us. But with the manifest file generation, it jumps to 350 seconds. And the incremental build times are also around 14-15 seconds. Let me know if i can help with any other information.

I tried to comment some parts of the code from the config file and try to find the bottleneck and i saw that this line in the config adds the most delay

            modules: [path.join(__dirname, '/src'), 'node_modules', 'src/pp/core/less/'],

Reproduce link

No response

Reproduce Steps

I can't provide a link to reproducible code since the code is private.

[glenjamin]

I also found adding this plugin added a significant slowdown to my compilation: ~87s instead of ~24s.

It also produces this warning

warning[internal]: custom stage for process_assets is not supported yet, so Infinity is fallback to Compilation.PROCESS_ASSETS_STAGE_REPORT(5000)

I had a look at what the plugin actually does under the hood, and it seems to be incredibly complicated. It seems odd that the webpack API doesn't appear to make it easy to match up entrypoints to the appropriate output files - the equivalent code in the HTML plugin is also quite involved.

I wonder if this is something that rspack could make easier to do via builtins?

edit: to follow-up on this, I've done a very basic inline plugin in my config which can produce a good-enough™ manifest for what we need - perhaps it'll help others too

import sources from 'webpack-sources';

/**
 * @type {import('@rspack/core').RspackPluginFunction}
 */
function AssetManifest(compiler) {
  const manifest = {};
  compiler.hooks.afterCompile.tap('asset-manifest', (compilation) => {
    const stats = compilation.getStats().toJson({
      all: false,
      entrypoints: true,
    });
    for (const entry of Object.values(stats.entrypoints || {})) {
      for (const asset of entry.assets) {
        manifest[unhashed(asset.name)] = asset.name;
      }
    }
    compilation.emitAsset(
      'manifest.json',
      new sources.RawSource(JSON.stringify(manifest, null, 2)),
    );
  });
}

/**
 * Strip the webpack [contenthash] from a filename via heuristics
 *
 * @param {string} full
 * @returns string
 */
function unhashed(full) {
  return full.replace(/\.[a-f0-9]{20}\./, '.');
}

[hardfist]

@glenjamin i think this plugin can be implemented in a easy way，like post-process stats.json just dont have time to investigate，can you give it a try

[glenjamin]

I couldn't find a nicer way to build a mapping of "clean" names to hashed names other than string manipulation on the final asset name - this seems to be basically what all of the existing plugins do anyway.

A basic version of the plugin isn't very difficult, as demonstrated by the snippet I posted above, but a more general version that's packaged up nicely might be harder, as the existing webpack asset plugins have a lot of configuration options for some reason.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If this issue is still affecting you, please leave any comment (for example, "bump"). We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[krutoo]

is this still an issue?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If this issue is still affecting you, please leave any comment (for example, "bump"). We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[krutoo]

Any updates?

[h-a-n-a]

@krutoo Try out our latest version and see if this still exists.

[LingyuCoder]

The current version has already made a lot of optimizations for the processing of assets, and the performance bottleneck in the internal project has been resolved, so this issue will be closed first. If you encounter similar problems, please reopen it.