Important CPU and memory usage

[vincentbernat]

Hi!

We upgraded our installation to the git version from 2013/10/22 without any other changes than updating the default tables. We observe an important grow in memory and CPU usage.

I think this is expected because of the new features but could we get any performance hint on how Pinba should be expected to perform memory/CPU wise. How is it related to the number of requests, the number of custom tables and their types. What should we watch if we want to reduce memory or CPU usage? Some recipes would be welcome.

[tony2001]

Yes, increased memory usage is expected, since now Pinba stores more information for each row in a report (see https://github.com/tony2001/pinba_engine/wiki/Reports#wiki-Histograms). I didn't notice any sizeable CPU usage increase here, though, so this is not expected.

What kind of data do you have? Do you have a lot of timers? What kind of reports? And how many of them?

[tony2001]

Also which version did you use previously?

[vincentbernat]

I was using 1.0.0. I have no request using the request table directly. Most CPU seems to be used when requests are running (we have a batch of requests every 30 seconds). Here are some info:

mysql> select * from info \G
*************************** 1. row ***************************
       req_count: 11170542
      time_total: 781834
  ru_utime_total: 436168
  ru_stime_total: 27108.4
   time_interval: 900
    kbytes_total: 173681000
memory_footprint: 0
 req_time_median: 0.0341832
     index_value: NULL
1 row in set (0.00 sec)

mysql> select count(*) from timer \G
*************************** 1. row ***************************
count(*): 52854822

mysql> select count(*) from timertag \G
*************************** 1. row ***************************
count(*): 70829224

I have put the non default tables in this gist: https://gist.github.com/7354644

Only tag_report_counter is returning many results (half a million).

[tony2001]

You might want to limit the time interval (pinba_stats_history in my.cnf) - that way you'll decrease both memory and CPU usage at once. 300 or 200 seconds is what I use mostly. Are you using Jemalloc or Hoard? They do help a lot, too. Both for memory and CPU usage.

[vincentbernat]

I am using jemalloc. I will reduce the time interval if I need to. There is currently no problem with CPU and memory usage. But since it is more important than it was previously, I wanted to share this with you.

[tony2001]

Ok, thanks for the info. The increased CPU usage might be also the result of heavy multithreading in Pinba, this way it uses existing resources much more effectively, not limiting itself to only one CPU core.

[vincentbernat]

OK, it seems that there are additional copies between temporary pools. Lowering the history length does not change CPU usage. But we can live with that.