Compaction threads - reasearch on changing the default values

[Guyme]

Changing the compactions threads from 1 to 8 showed dramatic performance changes especially in the write tests but not only. We would like a research to be done as to:

1. what is the advantage of more compaction threads.
2. what is the dis-advantage

We want to understand the theory behind it before we start the testing to pinpoint our testing efforts (and the release to which this will be active) accordingly.

here is the graph :

[Guyme]

Can we have a mini autotune to figure out what is the number of cores in the system and set this parameter based on it ?

[erez-speedb]

From this test comparing 4,8,16 threads on an 8 core instance https://admin.speedb.io/performance?items=83VJkkPPvXeEwL81dSql&items=oqNab7LLU1k8pfjxFalU&items=bZNRH5A4XaVrZlV4cud4&items=SEtSDriT1YcCX4kpvZHY&colors=%23F06292&colors=%239575CD&colors=%237986CB&colors=%2364B5F6 More compaction threads cost more memory and improve performance. Even 16 threads worked better than 4

[Guyme]

check on 1 or max 2 cores system and on multiple CF