voipmonitor
commented
7 years ago if you have the lock - what do you see in "show processlist" in mysql? Also - not a single cdr is written so it completely locks or inserting is only slow so the sql is rising?
We're noticing an issue since approx 19.3 where CDRs will stop writing to the SQL database. The SQLq counters increment for the C, C1, C2 and C3 variables, but no records appear to write down to MySQL.
We've tried a few config options, and also tried both MySQL and Percona, and tuned this. MySQL shows no process locks or deadlocks. The timing appears random, but it does seem like the locks may start under higher load.
Under normal load we'll see:
calls[237,r:172][237,r:172] PS[C:1/-0 r:17/-12 S:207/207 SR:71 SM:- R:25920 A:27460] SQLq[C:36480 C2:7799 C3:1610 M:0 R:0 Cl:0] heap[0|1|0] comp[42] [47.5Mb/s] tarQ[642] tarB[44MB] tarCPU[0.2|0.2|0.1|0.1|0.3|0.2%] t0i_ens3_CPU[4.4Mb/s;0.1%/2.3%] t0CPU[2.3%] t1CPU[0.1%/0.1%/0.8%/0.0%/0.2%] t2CPU[pb:2.8/d4.0/rm:3.4/rh:0.5/rd:3.6/S:14.2%] tRTP_CPU[16.6%/16.6m/1t] tsip_tcpCPU[8l|0/0s|0/0p] tacCPU[2.4%] RSS/VSZ[663|1312]MB LA[4.15 3.96 3.85] v19.5
but when it's not inserting we'll see:
Jun 8 15:50:23 voipmonitor voipmonitor[13185]: calls[191,r:94][191,r:94] PS[C:0/-1 r:12/-16 S:158/158 SR:50 SM:- R:20600 A:21720] SQLq[C:36403 C2:7722 C3:1532 M:0 R:0 Cl:0] heap[0|1|0] comp[42] [35.3Mb/s] tarQ[539] tarB[39MB] tarCPU[0.1|0.1|0.1|0.2|0.3|0.1|0.1%] t0i_ens3_CPU[3.1Mb/s;0.0%/2.6%] t0CPU[4.5%] t1CPU[0.1%/0.0%/0.7%/0.0%/0.2%] t2CPU[pb:1.6/d4.1/rm:2.5/rh:0.3/rd:2.6/S:11.2%] tRTP_CPU[11.7%/11.7m/1t] tsip_tcpCPU[9l|0/0s|0/0p] tacCPU[1.3%] RSS/VSZ[648|1312]MB LA[4.04 3.83 3.80] v19.5
The PCAPs appear to still write down.
Data is being sent across the mirror protocol from a number of distributed probes.
Currently testing develop b1e0ed3 and can see the issue, however have observed the issue with stock built 19.5, 19.4.2, 19.4.1, 19.3.
Base OS is Ubuntu 16 Linux voipmonitor 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux and we think we've ruled out OS issues. MySQL tables are healthy.