[Bug]: reading assets in the plugin takes a long time

[liangyuetian]

Version

System:
    OS: macOS 14.2.1
    CPU: (8) arm64 Apple M1 Pro
    Memory: 217.88 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Browsers:
    Chrome: 121.0.6167.184
    Safari: 17.2.1
  npmPackages:
    @rsbuild/core: ^0.4.3 => 0.4.3 
    @rsbuild/plugin-vue: ^0.4.3 => 0.4.3

Details

compiler.hooks.compilation.tap("RsBuildPlugin1", (compilation) => {
            compilation.hooks.processAssets.tap(
                {
                    name: 'MyPlugin',
                    stage: webpack.Compilation.PROCESS_ASSETS_STAGE_DERIVED, // see below for more stages
                },
                (assets) => {

                    const t1 = collectTime(() => {
                        const length = Object.entries(assets).length
                        return length
                    })
                    console.log(`1. count: ${t1[1]}; time: ${t1[0]}ms`)
                    const t2 = collectTime(() => {
                        const length = Object.entries(assets).length
                        return length
                    })
                    console.log(`2. count: ${t2[1]}; time: ${t2[0]}ms`)

                })
        })

Reproduce link

https://github.com/liangyuetian/rsbuild-hmr-slow-demo/blob/webpack_assets_long_time/rspack.plugin1.js#L24

Reproduce Steps

1. git checkout -b webpack_assets_long_time
2. npm run dev

[xc2]

thank you @liangyuetian

@chenjiahan could you pls move the issue to rspack?

[chenjiahan]

done~

[xc2]

In my computer (m2 max mac studio), it takes 14ms every time to access assets.x when x is a js asset.

@h-a-n-a

Whenever I access assets.x, it always involves retrieving the source of x from the rust side.

I believe this step is necessary because assets.x is being synchronized for read and write operations with the rust side before the processAssets and afterProcessAssets hooks are completed.

Therefore, I don't think rspack can improve the performance in this case at the moment, correct?

[h-a-n-a]

In my computer (m2 max mac studio), it takes 14ms every time to access assets.x when x is a js asset.

@h-a-n-a

Whenever I access assets.x, it always involves retrieving the source of x from the rust side.

I believe this step is necessary because assets.x is being synchronized for read and write operations with the rust side before the processAssets and afterProcessAssets hooks are completed.

Therefore, I don't think rspack can improve the performance in this case at the moment, correct?

@xc2

Currently, we try to mitigate the performance impact by using a thin proxy layer on compilation.assets, which is also the parameter of the hooks that've been mentioned above. So that means it's an on-demand FFI communication and it contains converting Rust struct into Node-API compatible way, and passing it to node, then recreate a webpack-source compatible Source struct.

When retrieving assets source from Rust, the recommended way is, if you know the exact name of it, to access with the name of the asset. For example, use assets.x, this only passes Source of x to JS. Also to note, the communication between Rust and Node side is necessary considering if there's some changes made to this asset before reading this again.

Object.entries(assets) is not recommended. This will pass every filename and source to JS side.

[liangyuetian]

In my computer (m2 max mac studio), it takes 14ms every time to access assets.x when x is a js asset. @h-a-n-a Whenever I access assets.x, it always involves retrieving the source of x from the rust side. I believe this step is necessary because assets.x is being synchronized for read and write operations with the rust side before the processAssets and afterProcessAssets hooks are completed. Therefore, I don't think rspack can improve the performance in this case at the moment, correct?

@xc2

Currently, we try to mitigate the performance impact by using a thin proxy layer on compilation.assets, which is also the parameter of the hooks that've been mentioned above. So that means it's an on-demand FFI communication and it contains converting Rust struct into Node-API compatible way, and passing it to node, then recreate a webpack-source compatible Source struct.

When retrieving assets source from Rust, the recommended way is, if you know the exact name of it, to access with the name of the asset. For example, use assets.x, this only passes Source of x to JS. Also to note, the communication between Rust and Node side is necessary considering if there's some changes made to this asset before reading this again.

Object.entries(assets) is not recommended. This will pass every filename and source to JS side.

The time consumption of a single FFI+ Node-API+ webpack source is also not short

In demo project https://github.com/liangyuetian/rsbuild-hmr-slow-demo/blob/webpack_assets_long_time/rspack.plugin1.js#L30

In real projects

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If this issue is still affecting you, please leave any comment (for example, "bump"). We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[h-a-n-a]

bump

[h-a-n-a]

In my computer (m2 max mac studio), it takes 14ms every time to access assets.x when x is a js asset. @h-a-n-a Whenever I access assets.x, it always involves retrieving the source of x from the rust side. I believe this step is necessary because assets.x is being synchronized for read and write operations with the rust side before the processAssets and afterProcessAssets hooks are completed. Therefore, I don't think rspack can improve the performance in this case at the moment, correct?

@xc2 Currently, we try to mitigate the performance impact by using a thin proxy layer on compilation.assets, which is also the parameter of the hooks that've been mentioned above. So that means it's an on-demand FFI communication and it contains converting Rust struct into Node-API compatible way, and passing it to node, then recreate a webpack-source compatible Source struct. When retrieving assets source from Rust, the recommended way is, if you know the exact name of it, to access with the name of the asset. For example, use assets.x, this only passes Source of x to JS. Also to note, the communication between Rust and Node side is necessary considering if there's some changes made to this asset before reading this again. Object.entries(assets) is not recommended. This will pass every filename and source to JS side.

The time consumption of a single FFI+ Node-API+ webpack source is also not short

In demo project https://github.com/liangyuetian/rsbuild-hmr-slow-demo/blob/webpack_assets_long_time/rspack.plugin1.js#L30

In real projects

@liangyuetian It would be nice to provide your real-world example to see if there's anything else we can do. The solution could be enqueue the changes in the same tick and flush them all at once. Feel free to ping me.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If this issue is still affecting you, please leave any comment (for example, "bump"). We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!