[WIP] All threads compiling parallelism scaling experiment

[NHDaly]

All tasks are compiling a new anonymous function lambda and then executing it (eval and invokelatest).

The goal is to measure scaling, to measure contention from compilation. The goal would be that compiling entirely independent code would have 0 contention.

In practice, it looks like currently it has 100% contention, due to a global mutex around compilation.

The code for this benchmark (added in https://github.com/RelationalAI-oss/MultithreadingBenchmarks.jl/commit/0f1c498fcaf7a81e1f03e200aa23ad837d04950a) is here: https://github.com/RelationalAI-oss/MultithreadingBenchmarks.jl/blob/0f1c498/src/bench/all_tasks_compiling.jl

[NHDaly]

Early Results: Roadblocks and future work

Currently, running this benchmark fails, because too many parallel tasks compiling seems to trigger a race-condition in julia, as detailed here: https://github.com/JuliaLang/julia/issues/33183

However, profiling shows that currently there is essentially 100% contention if all threads are compiling, because there is a global lock on compilation:

Some related discussion on this from slack:

@nhdaly: I don't think we would need to hold a global mutex during compilation, right? Is this on the to-do list for future work for multithreading? @staticfloat: Is LLVM itself properly multithreaded? I could envision the compilation pipeline itself being made asynchronous (e.g. parsing, Julia optimization, LLVM optimization, LLVM codegen; each stage of compilation running independently of the rest, but within each stage, running serially) but I'm not sure that LLVM itself runs in a multithreaded fashion. That being said, I'm pretty sure a huge chunk of our time is spent in type inference, so maybe there are still large speedups to be had @nhdaly: Thanks Elliot, yeah that makes sense. I know that this kind of "slow code" has explicitly not been a priority for the multithreading team so far, so I'm not complaining, just exploring the current status. :slightly_smiling_face: Ooh, yeah, i hadn't considered whether LLVM itself could be run simultaneously from multiple threads... :cry: I would not be surprised if it isn't properly multithreaded. @vchuravy: LLVM can be multithreaded there has been a lot of work in that area recently

So there would need to be investigations on work done to both:

• [ ] Allow parallel invocations of the julia compilation pipeline (parsing, lowering, type inference, julia optimization, etc)
• [ ] Enable a multithreaded LLVM to allow parallel invocations of the LLVM part (LLVM optimization, LLVM codegen)