Very high CPU usage when accessing BufferPool

[pliard-b]

We have a Rust service that has thousands of threads using their own mysql::Conn to multiple databases. We noticed up to 10x CPU usage increases when going from mysql v20 to v21. The regression appears to be caused by contention on the added global buffer pool and the implementation of std::sync::Mutex involving spinlocks (see below). FYI we are using jemalloc and the Vec allocations that the usage of BufferPool replaced didn't appear to be a bottleneck for us.

Here are some suggestions that I can think of for how this can be addressed: 1) Revert to allocating Vec's on demand

• Pros:
  • Known state; this is what was used prior to the introduction of BufferPool
  • Minimizes memory usage
  • Performs well enough with "good" heap allocators
• Cons:
  • Might not perform so well with e.g. glibc's malloc

2) Make each mysq::Conn object have its own buffer

• Pros:
  • Minimizes heap allocations
• Cons:
  • Likely increases memory usage as now each connection keeps a long-lived buffer

3) Have BufferPool use a lock-free pool data structure such as what the pool crate provides:

• Pros:
  • Minimal code changes
  • Should perform better than current approach under high concurrency
  • Minimizes heap allocations
• Cons:
  • May not fully address the CPU usage regression depending on how well pool or the chosen alternative performs
  • Adds another dependency
  • Still involves global state which is not ideal
  • Might cause higher memory usage than 1)

In addition to these a hybrid approach could be used where a new flag controls whether 1-2) or 3) are used.

[blackbeam]

Hi. This might be addressed in blackbeam/mysql_async#170. I'm planning to port this for the next release.

Also I think that a flag that turns of the pool is the best option for your case, so I'll add it.

[pliard-b]

Thanks a lot @blackbeam for fixing this. We will give the pool another try with the crossbeam ArrayQueue. Disabling the pool through the environment variable seems a bit fragile though. Perhaps this could be done in the future via a cargo feature?

[blackbeam]

Disabling the pool through the environment variable seems a bit fragile though. Perhaps this could be done in the future via a cargo feature?

Agree. This is now addressed and will be in v22.0.0 release.

[pliard-b]

Disabling the pool through the environment variable seems a bit fragile though. Perhaps this could be done in the future via a cargo feature?

Agree. This is now addressed and will be in v22.0.0 release.

Great, thank you!