Evaluate and improve the Config Compiler performance

[dnsmichi]

Tasks

Focus on the compiler, not the lexer/parser parts.

• [ ] Evaluate whether future/promise works within the config compiler stages
• [ ] Measure the impact with std::shuffle and ParallelFor inside work queues and evaluate other possible algorithms.
  • Especially ParallelFor involves a WQ and threads which may lock themselves.
• [ ] Check whether we can decrease the memory footprint even more
• [ ] Analyse the code and try to make it more readable

These steps may result into specific sub issues, and should be iterative on the matter.

Story

In the past, we had an implementation which used future/promise design patterns. This was reverted due to the fact that too many open file handles were there. The performance was better back then, I don't remember the numbers exactly.

With the exceeding numbers of monitoring objects and thus longer config compilation times, we should evaluate possible improvements here.

Keep in mind that the config compiler stages add some dependencies how configurations work. Changing the Commit, PreActivate, etc. stages also changes dependencies within the object activation and what not. Consider this the last resort going there.

[Al2Klimov]

It seems we can't also eval the compiled expressions in parallel that easy as we're using thread-local vars in some expressions. For a possible solution look for the VThread struct in the sandbox.

[dnsmichi]

It seems we can't also eval the compiled expressions in parallel that easy as we're using thread-local vars in some expressions.

Can you please share some concrete examples for the core devs not so deep into this yet?

[Al2Klimov]

Config objects, apply rules and #7781.

[Al2Klimov]

git ls-files -z |xargs -0 grep --color=always -nFwe thread_local -e thread_specific_ptr

[Al2Klimov]

Maybe it's not that hard, but you could get deadlocks while one thread in the full threadpool waits for others to complete.