extremely high number of kworkers

[disaster123]

Since a few weeks i'm seeing high spikes of kworkers on the systems running with your patches.

Normally those systems have around 60 kworkers. From time to time this goes up to 2500 kworker processes (for around 5 minutes).

[ ~]# ps aux|grep kworker | wc -l 2452

[ ~]# ps aux|grep kworker | wc -l 59

List looks like this:

root 11077 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:143] root 11078 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:144] root 11079 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:145] root 11080 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:146] root 11081 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:147] root 11082 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:148] root 11083 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:149] root 11084 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:150] root 11085 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:151] root 11086 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:152] root 11087 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:153] root 11088 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:154] root 11089 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:155] root 11090 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:156] root 11091 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:157] root 11092 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:158] root 11093 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:159] root 11094 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:160] root 11095 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:161] root 11098 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:162] root 11099 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:163] root 11100 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:164] root 11101 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:165] root 11102 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:166] root 11103 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:167] root 11104 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:168] root 11105 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:169] root 11106 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:170] root 11107 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:171] root 11108 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:172] root 11109 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:173] root 11110 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:174] root 11111 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:175] root 11112 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:176] root 11113 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:177] root 11114 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:178] root 11115 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:179] root 11116 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:180] root 11117 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:181] root 11118 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:182] root 11119 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:183] root 11120 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:184] root 11121 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:185] root 11127 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:186] root 11128 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:187] root 11130 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:188] root 11136 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:189] root 11137 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:190] root 11138 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:191] root 11140 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:192] root 11141 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:193] root 11142 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:194] root 11143 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:195] root 11144 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:196] root 11145 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:197] root 11146 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:198] root 11147 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:199] root 11148 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:200] root 11149 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:201] root 11150 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:202] root 11151 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:203] root 11152 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:204] root 11153 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:205] root 11154 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:206] root 11155 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:207] root 11156 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:208] root 11157 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:209] root 11158 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:210] root 11159 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:211] root 11160 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:212] root 11161 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:213] root 11162 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:214] root 11163 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:215] root 11164 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:216] root 11165 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:217] root 11166 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:218] root 11167 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:219] root 11168 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:220] root 11169 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/11:221] root 11174 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:0] root 11175 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:3] root 11176 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:4] root 11177 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:5] root 11178 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:6] root 11179 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:7] root 11180 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:8] root 11181 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:9] root 11182 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:10] root 11183 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:11] root 11184 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:12] root 11185 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:13] root 11186 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:14] root 11187 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:15] root 11188 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:16] root 11189 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:17] root 11190 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:18] root 11191 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:19] root 11192 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:20] root 11193 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:21] root 11194 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:22] root 11195 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:23] root 11196 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:24] root 11197 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:25] root 11198 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:26] root 11199 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:27] root 11200 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:28] root 11201 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:29] root 11202 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:30] root 11203 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:31] root 11204 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:32] root 11205 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:33] root 11206 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:34] root 11207 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:35] root 11208 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:36] root 11209 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:37] root 11210 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:38] root 11211 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:39] root 11212 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:40] root 11213 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:41] root 11214 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:42] root 11215 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:43] root 11216 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:44] root 11217 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:45] root 11218 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:46] root 11219 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:47] root 11220 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:48] root 11221 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:49] root 11222 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:50] root 11223 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:51] root 11224 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:52] root 11225 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:53] root 11226 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:54] root 11227 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:55] root 11228 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:56] root 11229 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:57] root 11230 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:58] root 11231 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:59] root 11232 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:60] root 11233 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:61] root 11234 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:62] root 11235 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:63] root 11236 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:64] root 11237 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:65] root 11238 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:66] root 11239 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:67] root 11240 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:68] root 11241 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:69] root 11242 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:70] root 11243 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:71] root 11244 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:72] root 11245 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:73] root 11246 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:74] root 11247 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:75] root 11248 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:76] root 11249 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:77] root 11250 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:78] root 11251 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:79] root 11252 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:80] root 11253 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:81] root 11254 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:82] root 11255 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:83] root 11256 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:84] root 11257 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:85] root 11258 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:86] root 11259 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:87] root 11260 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:88] root 11261 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:89] root 11262 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:90] root 11263 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:91] root 11264 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:92] root 11265 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:93] root 11266 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:94] root 11267 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:95] root 11268 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:96] root 11269 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:97] root 11270 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:98] root 11271 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:99] root 11272 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:100] root 11273 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:101] root 11274 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:102] root 11275 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:103] root 11276 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:104] root 11277 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:105] root 11278 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:106] root 11279 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:107] root 11280 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:108] root 11281 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:109] root 11282 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:110] root 11283 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:111] root 11284 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:112] root 11285 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:113] root 11286 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:114] root 11287 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:115] root 11288 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:116] root 11289 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:117] root 11290 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:118] root 11291 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:119] root 11292 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:120] root 11293 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:121] root 11294 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:122] root 11295 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:123] root 11296 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:124] root 11297 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:125] root 11298 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:126] root 11299 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:127] root 11300 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:128] root 11301 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:129] root 11305 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:130] root 11306 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:131] root 11307 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:132] root 11308 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:133] root 11309 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:134] root 11314 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:135] root 11315 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:136] root 11316 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:137] root 11317 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:138] root 11318 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:139] root 11319 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:140] root 11320 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:141] root 11321 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:142] root 11322 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:143] root 11323 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:144] root 11324 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:145] root 11325 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:146] root 11326 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:147] root 11328 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:148] root 11329 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:149] root 11330 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:150] root 11331 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:151] root 11333 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:152] root 11334 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:153] root 11335 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:154] root 11336 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:155] root 11337 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:156] root 11338 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:157] root 11339 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:158] root 11341 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:159] root 11342 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:160] root 11345 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:161] root 11347 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:162] root 11348 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:163] root 11349 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:164] root 11350 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:165] root 11352 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:166] root 11353 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:167] root 11354 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:168] root 11356 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:169] root 11357 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:170] root 11360 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:171] root 11361 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:172] root 11362 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:173] root 11363 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:174] root 11365 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:175] root 11366 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:176] root 11367 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:177] root 11369 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:178] root 11370 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:179] root 11371 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:180] root 11372 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:181] root 11373 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:182] root 11374 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:183] root 11376 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:184] root 11377 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:185] root 11378 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:186] root 11379 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:187] root 11380 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:188] root 11381 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:189] root 11382 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:190] root 11383 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:191] root 11385 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:192] root 11387 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:193] root 11388 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:194] root 11389 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:195] root 11390 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:196] root 11391 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:197] root 11392 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:198] root 11393 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:199] root 11394 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:200] root 11395 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:201] root 11396 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:202] root 11397 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:203] root 11398 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:204] root 11399 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:205] root 11400 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:206] root 11401 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:207] root 11402 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:208] root 11403 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:209] root 11404 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:210] root 11405 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:211] root 11406 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:212] root 11407 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:213] root 11408 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:214] root 11409 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:215] root 11410 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:216] root 11411 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:217] root 11412 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:218] root 11413 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:219] root 11414 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:220] root 11415 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:221] root 11416 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:222] root 11417 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:223] root 11418 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:224] root 11419 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:225] root 11422 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:226] root 11423 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/9:227] root 11435 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:221] root 11436 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:222] root 11439 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:223] root 11440 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:224] root 11441 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:225] root 11442 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:226] root 11443 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:227] root 11444 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:228] root 11445 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/1:229] root 11449 0.1 0.0 0 0 ? S 10:45 0:00 [kworker/u24:1] root 11450 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:44] root 11451 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:45] root 11452 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:46] root 11453 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:47] root 11454 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:48] root 11455 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:49] root 11456 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:50] root 11457 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:51] root 11458 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:52] root 11459 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:53] root 11460 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:54] root 11461 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:55] root 11462 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:56] root 11463 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:57] root 11464 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:58] root 11465 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:59] root 11466 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:60] root 11467 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:61] root 11468 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:62] root 11469 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:63] root 11470 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:64] root 11471 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:65] root 11472 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:66] root 11473 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:67] root 11474 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:68] root 11475 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:69] root 11476 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:70] root 11477 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:71] root 11478 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:72] root 11479 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:73] root 11480 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:74] root 11481 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:75] root 11482 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:76] root 11483 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:77] root 11484 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:78] root 11485 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:79] root 11486 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:80] root 11487 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:81] root 11488 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:82] root 11489 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:83] root 11490 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:84] root 11491 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:85] root 11492 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:86] root 11493 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:87] root 11494 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:88] root 11495 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:89] root 11496 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:90] root 11497 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:91] root 11498 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:92] root 11499 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:93] root 11500 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:94] root 11501 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:95] root 11502 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:96] root 11503 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:97] root 11504 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:98] root 11505 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:99] root 11506 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:100] root 11507 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:101] root 11508 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:102] root 11509 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:103] root 11510 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:104] root 11511 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:105] root 11512 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:106] root 11513 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:107] root 11514 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:108] root 11515 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:109] root 11516 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:110] root 11517 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:111] root 11518 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:112] root 11519 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:113] root 11522 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:114] root 11523 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:115] root 11524 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:116] root 11525 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:117] root 11526 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:118] root 11527 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:119] root 11528 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:120] root 11529 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:121] root 11530 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:122] root 11531 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:123] root 11532 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:124] root 11533 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:125] root 11534 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:126] root 11535 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:127] root 11536 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:128] root 11537 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:129] root 11538 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:130] root 11539 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:131] root 11540 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:132] root 11541 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:133] root 11542 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:134] root 11543 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:135] root 11544 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:136] root 11545 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:137] root 11546 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:138] root 11547 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:139] root 11548 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:140] root 11549 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:141] root 11550 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:142] root 11551 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:143] root 11552 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:144] root 11553 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:145] root 11554 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:146] root 11555 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:147] root 11556 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:148] root 11557 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:149] root 11558 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:150] root 11559 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:151] root 11560 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:152] root 11561 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:153] root 11562 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:154] root 11563 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:155] root 11564 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:156] root 11565 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:157] root 11566 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:158] root 11567 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:159] root 11568 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:160] root 11569 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:161] root 11570 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:162] root 11571 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:163] root 11572 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:164] root 11573 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:165] root 11576 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:166] root 11577 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:167] root 11578 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:168] root 11579 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:169] root 11580 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:170] root 11581 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:171] root 11582 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:172] root 11583 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:173] root 11584 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:174] root 11585 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:175] root 11586 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:176] root 11587 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:177] root 11588 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:178] root 11589 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:179] root 11590 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:180] root 11591 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:181] root 11592 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:182] root 11593 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:183] root 11594 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:184] root 11595 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:185] root 11596 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:186] root 11597 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:187] root 11598 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:188] root 11599 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:189] root 11600 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:190] root 11601 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:191] root 11602 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:192] root 11603 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:193] root 11604 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:194] root 11605 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:195] root 11606 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:196] root 11607 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:197] root 11608 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:198] root 11609 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:199] root 11610 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:200] root 11613 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:201] root 11614 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:202] root 11615 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:203] root 11616 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:204] root 11617 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:205] root 11618 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:206] root 11619 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:207] root 11620 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:208] root 11621 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:209] root 11622 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:210] root 11623 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:211] root 11624 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:212] root 11625 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:213] root 11626 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:214] root 11627 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:215] root 11628 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:216] root 11629 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:217] root 11630 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:218] root 11631 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:219] root 11632 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:220] root 11633 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:221] root 11634 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:222] root 11635 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:223] root 11636 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:224] root 11637 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:225] root 11638 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:226] root 11641 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:227] root 11642 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:228] root 11645 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:229] root 11646 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:230] root 11647 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:231] root 11657 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:232] root 11658 0.0 0.0 0 0 ? S 10:45 0:00 [kworker/5:233] root 11669 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/2:235] root 11670 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/2:236] root 11678 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:230] root 11679 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:231] root 11680 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:232] root 11750 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:233] root 11751 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:234] root 11752 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:235] root 11755 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:236] root 11756 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:237] root 11757 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:238] root 11758 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:239] root 11759 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:240] root 11760 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:241] root 11761 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:242] root 11762 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:243] root 11763 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:244] root 11764 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:245] root 11765 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:246] root 11766 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:247] root 11767 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:248] root 11768 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:249] root 11769 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:250] root 11770 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:251] root 11771 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:252] root 11772 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:253] root 11773 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:254] root 11774 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:255] root 11775 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:256] root 11776 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/1:257] root 11778 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/5:234] root 11788 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:223] root 11791 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:224] root 11792 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:225] root 11793 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:226] root 11794 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:227] root 11795 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:228] root 11796 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:229] root 11797 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:230] root 11798 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:231] root 11799 0.0 0.0 0 0 ? S 10:46 0:00 [kworker/4:232] root 12345 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/0:0H] root 15923 0.0 0.0 0 0 ? S 04:47 0:00 [kworker/10:2] root 17175 0.0 0.0 0 0 ? S 09:33 0:00 [kworker/5:2] root 17489 0.0 0.0 0 0 ? S May12 0:00 [kworker/9:2] root 22514 0.0 0.0 0 0 ? S 03:29 0:02 [kworker/1:4] root 24188 0.0 0.0 0 0 ? S May11 0:12 [kworker/3:0] root 26023 0.0 0.0 0 0 ? S 09:56 0:00 [kworker/7:2] root 26668 0.0 0.0 0 0 ? S 09:59 0:00 [kworker/6:1] root 28512 0.0 0.0 0 0 ? S May12 0:00 [kworker/11:0] root 28994 0.0 0.0 0 0 ? S 10:06 0:01 [kworker/u24:0]

[hhoffstaette]

When this happens issue a echo 1 > /proc/sysrq-trigger to get backtraces. Alternatively you can try to figure out why so many threads are created in the first place, this explains how. The resulting trace file should show what causes the workqueues to bloat.

[hhoffstaette]

I have looked over all recent patches and not found anything that could be directly responsible, other than generic btrfs overload (compress workers?). As to why they are created instead of being reused - no idea.

In any case the fact that these kworkers go away after a while is not surprising as they seem to be idle, and 5 minutes is the default idle cleanup period.

I have one more suspicion (3dd1b67758 aka 'btrfs: improve compression workspaces memory management') which interacts with the compress functions, but have not yet read through them if this could really be the case. Since those patches don't fix a critical bug you could try without them and see whether the number of kworkers still spikes.

As last resort try to patch workqueue.c:IDLE_WORKER_TIMEOUT to something much lower, like 10-30 seconds or so. :-)

[disaster123]

Sorry this one isn't using btrfs at all. It heavily uses XFS.

[hhoffstaette]

Sorry this one isn't using btrfs at all. It heavily uses XFS.

That would have been great to know up front, before I spent a few hours digging. :(

I have no idea why the XFS patches (all from 4.5/4.6 except for some that were too risky) would cause this; I use all this on my own server & workstation 24/7, and have ~35 kworkers at most. Not as many cores as you, though. Since I have no information what these kworkers are/were doing it could also be a regression that will show up with 4.5/4.6.

Just to make sure I just ran iozone -a on my XFS volume in all combinations of normal/direct/aio and have no kthread bloat whatsoever, no matter how many aio ops I start.

If you really want you can try to remove xfs-* and try that (no format changes/compatibility problems), and that would give you XFS as in plain 4.4.x. Until then I just don't know.

If this is a Ceph OSD then it could just be crazy aio or something in Ceph.

[disaster123]

i'm sorry i did not see a link to FS behaviour and didn't thought about FS in this case at all.

I'll try to remove all xfs stuff and see what's happening.

[hhoffstaette]

While it would be good to know what happens without xfs-*, it's probably more useful to try with the latest vanilla 4.6-rc instead - simply to get a consistent way to report things. I also just read your latest mail to the XFS list, and there are so many other things getting mixed up both here and there - including your use of memory control groups (memcgs), which are probably still half-broken in 4.4 - that it's hard to give you any meaningful help. I'm trying really hard to not break anything (mostly since I use these patches for myself) but if you need real support, it would probably be best to talk to RedHat or SuSE. I'm closing this since I don't see how I can help you.

[disaster123]

Thx anyway - but i don't use memcgs - no idea where you git this. Also the problems to XFS list is without your patches and plain vanilla 4.4.10.