search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.85k stars 677 forks source link

Wasm runtime compilation is largely single-threaded #2636

Open nazar-pc opened 10 months ago

nazar-pc commented 10 months ago

Is there an existing issue?

Experiencing problems? Have you tried our Stack Exchange first?

Description of bug

I was recompiling things on Windows many times last few days and noticed that wasm runtime build script only uses a single CPU core for some reason, slowing things down significantly.

I'm not sure why this happens, but I don't think I have seen this behavior on Linux.

This is not a priority, but maybe someone knows why this happens and can point me in the direction that will help to resolve this.

Steps to reproduce

No response

bkchr commented 10 months ago

What profile did you use? Release?

nazar-pc commented 10 months ago

Yes, standard release profile

nazar-pc commented 2 months ago

Clarification: it is largely single-threaded even on Linux

bkchr commented 2 months ago

https://github.com/paritytech/polkadot-sdk/pull/4946 there was this pr recently to ensure we are not overloading the system.

nazar-pc commented 2 months ago

Well, it is massively under-utilizing my system instead. On 32C64T system seeing single remaining WASM runtime blocking other crates from being compiled while using a single CPU core is a frustrating experience. It is the most pronounced for polkadot-sdk contributions where there are many crates including numerous runtimes to compile.

bkchr commented 2 months ago

On 32C64T system seeing single remaining WASM runtime blocking other crates from being compiled while using a single CPU core is a frustrating experience

Ahh now I get what you mean. These are basically pipeline blockers, forcing the compiler to wait on them to compile. When it comes to linking, compression etc it will not use multiple CPUs anymore.

bkchr commented 2 months ago

Not sure we can do that much. I mean we can probably move some crates around, but in the grand scheme I doubt that it will change something.

nazar-pc commented 2 months ago

I understand fundamental constraints of linking and such, but build script of runtimes takes incomparably large amount of time for some reason, there must be some low-hanging fruits there. Compiling native runtime takes a small fraction of the time.