Closed jchodera closed 8 years ago
It looks like the multiprocessing version can keep a lot of processes hanging. After a couple days of running, I noted 211 processes:
multiprocessing
-bash-4.1$ ps xauwww | grep python server 3259 0.0 0.0 103308 876 pts/0 S+ 18:40 0:00 grep python server 4000 1.6 0.0 1071708 203424 pts/2 Sl+ Mar05 24:25 python scripts/munge_fah_data_parallel.py server 4001 2.2 0.0 1094348 227188 pts/2 Sl+ Mar05 32:36 python scripts/munge_fah_data_parallel.py server 4002 1.4 0.0 1074172 206636 pts/2 Sl+ Mar05 21:49 python scripts/munge_fah_data_parallel.py server 4004 1.5 0.0 1072044 204540 pts/2 Sl+ Mar05 22:57 python scripts/munge_fah_data_parallel.py server 4005 1.5 0.0 1097284 229792 pts/2 Sl+ Mar05 22:37 python scripts/munge_fah_data_parallel.py server 4006 1.5 0.0 1073412 205988 pts/2 Sl+ Mar05 22:11 python scripts/munge_fah_data_parallel.py server 4007 1.5 0.0 1095004 227732 pts/2 Sl+ Mar05 22:48 python scripts/munge_fah_data_parallel.py server 4008 1.8 0.0 1072780 205260 pts/2 Sl+ Mar05 27:42 python scripts/munge_fah_data_parallel.py server 4009 1.6 0.0 1041116 173764 pts/2 Sl+ Mar05 24:21 python scripts/munge_fah_data_parallel.py server 4010 1.5 0.0 1045248 177592 pts/2 Sl+ Mar05 23:18 python scripts/munge_fah_data_parallel.py server 4011 1.7 0.0 1070764 202980 pts/2 Sl+ Mar05 25:52 python scripts/munge_fah_data_parallel.py server 4012 1.5 0.0 1092928 225484 pts/2 Sl+ Mar05 23:16 python scripts/munge_fah_data_parallel.py server 4014 2.1 0.0 1128196 260920 pts/2 Sl+ Mar05 32:02 python scripts/munge_fah_data_parallel.py server 4015 1.8 0.0 1072684 205312 pts/2 Sl+ Mar05 26:43 python scripts/munge_fah_data_parallel.py server 4016 1.7 0.0 1071136 203656 pts/2 Sl+ Mar05 25:19 python scripts/munge_fah_data_parallel.py server 4017 1.7 0.0 1072288 204720 pts/2 Sl+ Mar05 25:45 python scripts/munge_fah_data_parallel.py root 4077 0.0 0.0 205956 9496 ? S Mar01 0:23 /usr/bin/python -s /usr/sbin/osad --pid-file /var/run/osad.pid server 7678 1.7 0.0 1272640 208872 pts/2 Sl+ Mar05 25:34 python scripts/munge_fah_data_parallel.py server 7679 1.7 0.0 1308740 244968 pts/2 Sl+ Mar05 25:52 python scripts/munge_fah_data_parallel.py server 7680 1.8 0.0 1300404 236620 pts/2 Sl+ Mar05 26:06 python scripts/munge_fah_data_parallel.py server 7681 1.7 0.1 1337600 274132 pts/2 Sl+ Mar05 25:49 python scripts/munge_fah_data_parallel.py server 7682 1.7 0.0 1282828 218876 pts/2 Sl+ Mar05 25:52 python scripts/munge_fah_data_parallel.py server 7683 1.7 0.1 1330816 267356 pts/2 Sl+ Mar05 25:52 python scripts/munge_fah_data_parallel.py server 7684 1.7 0.0 1300216 236424 pts/2 Sl+ Mar05 25:38 python scripts/munge_fah_data_parallel.py server 7685 1.7 0.0 1275088 211172 pts/2 Sl+ Mar05 25:27 python scripts/munge_fah_data_parallel.py server 7686 1.7 0.0 1285176 221404 pts/2 Sl+ Mar05 25:48 python scripts/munge_fah_data_parallel.py server 7687 1.7 0.0 1292628 228840 pts/2 Sl+ Mar05 25:45 python scripts/munge_fah_data_parallel.py server 7688 1.7 0.0 1293104 229292 pts/2 Sl+ Mar05 25:25 python scripts/munge_fah_data_parallel.py server 7689 1.7 0.0 1276316 212396 pts/2 Sl+ Mar05 25:37 python scripts/munge_fah_data_parallel.py server 7690 1.7 0.0 1295628 232160 pts/2 Sl+ Mar05 25:46 python scripts/munge_fah_data_parallel.py server 7691 1.7 0.0 1318016 254548 pts/2 Sl+ Mar05 25:49 python scripts/munge_fah_data_parallel.py server 7692 2.0 0.0 1321140 257672 pts/2 Sl+ Mar05 28:50 python scripts/munge_fah_data_parallel.py server 7693 1.7 0.0 1269232 205440 pts/2 Sl+ Mar05 25:26 python scripts/munge_fah_data_parallel.py server 8063 1.5 0.0 2833624 197200 pts/2 Sl+ 01:17 16:34 python scripts/munge_fah_data_parallel.py server 8064 1.6 0.0 2833512 196980 pts/2 Sl+ 01:17 16:51 python scripts/munge_fah_data_parallel.py server 8065 1.6 0.0 2833368 196944 pts/2 Sl+ 01:17 16:52 python scripts/munge_fah_data_parallel.py server 8066 1.5 0.0 2833404 197056 pts/2 Sl+ 01:17 16:22 python scripts/munge_fah_data_parallel.py server 8067 1.5 0.0 2833736 197184 pts/2 Sl+ 01:17 16:27 python scripts/munge_fah_data_parallel.py server 8068 1.5 0.0 2833744 197192 pts/2 Sl+ 01:17 16:31 python scripts/munge_fah_data_parallel.py server 8069 1.4 0.0 2833092 196536 pts/2 Sl+ 01:17 15:35 python scripts/munge_fah_data_parallel.py server 8070 1.6 0.0 2833704 197152 pts/2 Sl+ 01:17 16:46 python scripts/munge_fah_data_parallel.py server 8071 1.3 0.0 2833736 197200 pts/2 Sl+ 01:17 13:49 python scripts/munge_fah_data_parallel.py server 8072 1.5 0.0 2833608 197052 pts/2 Sl+ 01:17 16:38 python scripts/munge_fah_data_parallel.py server 8073 1.5 0.0 2833700 197160 pts/2 Sl+ 01:17 16:40 python scripts/munge_fah_data_parallel.py server 8074 1.5 0.0 2840904 204528 pts/2 Sl+ 01:17 16:38 python scripts/munge_fah_data_parallel.py server 8075 1.5 0.0 2833588 197164 pts/2 Sl+ 01:17 16:40 python scripts/munge_fah_data_parallel.py server 8076 1.5 0.0 2833340 196920 pts/2 Sl+ 01:17 16:37 python scripts/munge_fah_data_parallel.py server 8077 1.5 0.0 2833724 197172 pts/2 Sl+ 01:17 16:37 python scripts/munge_fah_data_parallel.py server 8078 1.6 0.0 2833712 197156 pts/2 Sl+ 01:17 16:49 python scripts/munge_fah_data_parallel.py server 11172 98.4 0.0 3041824 209200 pts/2 Sl+ 01:38 1006:42 python scripts/munge_fah_data_parallel.py server 11173 98.5 0.1 3105724 273156 pts/2 Sl+ 01:38 1007:30 python scripts/munge_fah_data_parallel.py server 11174 98.9 0.0 3047860 215256 pts/2 Sl+ 01:38 1011:15 python scripts/munge_fah_data_parallel.py server 11175 98.4 0.0 3044124 211484 pts/2 Sl+ 01:38 1006:39 python scripts/munge_fah_data_parallel.py server 11176 99.4 0.1 3104544 271956 pts/2 Rl+ 01:38 1016:58 python scripts/munge_fah_data_parallel.py server 11177 99.2 0.1 3103288 270700 pts/2 Sl+ 01:38 1014:41 python scripts/munge_fah_data_parallel.py server 11178 99.0 0.1 3108832 276248 pts/2 Sl+ 01:38 1012:20 python scripts/munge_fah_data_parallel.py server 11179 98.8 0.0 3088436 255620 pts/2 Sl+ 01:38 1010:42 python scripts/munge_fah_data_parallel.py server 11180 98.5 0.0 3042332 209728 pts/2 Sl+ 01:38 1006:58 python scripts/munge_fah_data_parallel.py server 11181 99.5 0.1 3105416 272756 pts/2 Rl+ 01:38 1017:05 python scripts/munge_fah_data_parallel.py server 11182 99.4 0.1 3102796 270180 pts/2 Rl+ 01:38 1016:57 python scripts/munge_fah_data_parallel.py server 11183 98.7 0.1 3104076 271556 pts/2 Sl+ 01:38 1009:04 python scripts/munge_fah_data_parallel.py server 11184 98.8 0.0 3071572 239132 pts/2 Sl+ 01:38 1010:49 python scripts/munge_fah_data_parallel.py server 11185 98.4 0.0 3044320 211712 pts/2 Sl+ 01:38 1006:07 python scripts/munge_fah_data_parallel.py server 11186 99.0 0.1 3108496 275904 pts/2 Sl+ 01:38 1012:14 python scripts/munge_fah_data_parallel.py server 11187 99.5 0.0 3091052 258472 pts/2 Rl+ 01:38 1017:01 python scripts/munge_fah_data_parallel.py server 17278 1.8 0.0 1477932 219468 pts/2 Sl+ Mar05 26:02 python scripts/munge_fah_data_parallel.py server 17279 1.9 0.0 1478372 219748 pts/2 Sl+ Mar05 27:02 python scripts/munge_fah_data_parallel.py server 17280 1.7 0.0 1480480 221924 pts/2 Sl+ Mar05 24:49 python scripts/munge_fah_data_parallel.py server 17281 1.8 0.1 1537432 278576 pts/2 Sl+ Mar05 25:09 python scripts/munge_fah_data_parallel.py server 17282 1.7 0.1 1536932 278296 pts/2 Sl+ Mar05 25:02 python scripts/munge_fah_data_parallel.py server 17283 1.8 0.0 1478136 219684 pts/2 Sl+ Mar05 25:58 python scripts/munge_fah_data_parallel.py server 17284 1.8 0.1 1557156 298544 pts/2 Sl+ Mar05 25:59 python scripts/munge_fah_data_parallel.py server 17285 1.8 0.0 1477584 219112 pts/2 Sl+ Mar05 25:17 python scripts/munge_fah_data_parallel.py server 17286 1.7 0.0 1478208 219372 pts/2 Sl+ Mar05 24:56 python scripts/munge_fah_data_parallel.py server 17287 2.1 0.1 1539100 280744 pts/2 Sl+ Mar05 29:26 python scripts/munge_fah_data_parallel.py server 17288 1.7 0.0 1478864 220416 pts/2 Sl+ Mar05 24:23 python scripts/munge_fah_data_parallel.py server 17289 1.7 0.0 1479892 221048 pts/2 Sl+ Mar05 24:31 python scripts/munge_fah_data_parallel.py server 17290 1.9 0.0 1469212 210876 pts/2 Sl+ Mar05 27:43 python scripts/munge_fah_data_parallel.py server 17291 1.7 0.0 1479160 220288 pts/2 Sl+ Mar05 25:02 python scripts/munge_fah_data_parallel.py server 17292 1.9 0.0 1480544 221676 pts/2 Sl+ Mar05 27:21 python scripts/munge_fah_data_parallel.py server 17293 1.7 0.0 1477272 218384 pts/2 Sl+ Mar05 24:59 python scripts/munge_fah_data_parallel.py server 20326 1.4 0.1 1721736 264584 pts/2 Sl+ Mar05 19:51 python scripts/munge_fah_data_parallel.py server 20327 1.3 0.0 1701368 244204 pts/2 Sl+ Mar05 18:34 python scripts/munge_fah_data_parallel.py server 20328 1.3 0.0 1693204 236080 pts/2 Sl+ Mar05 18:26 python scripts/munge_fah_data_parallel.py server 20329 1.3 0.0 1685800 228680 pts/2 Sl+ Mar05 18:53 python scripts/munge_fah_data_parallel.py server 20330 1.4 0.0 1700328 243268 pts/2 Sl+ Mar05 20:16 python scripts/munge_fah_data_parallel.py server 20331 1.3 0.0 1709288 252184 pts/2 Sl+ Mar05 18:33 python scripts/munge_fah_data_parallel.py server 20332 1.3 0.0 1684236 227120 pts/2 Sl+ Mar05 19:08 python scripts/munge_fah_data_parallel.py server 20333 1.3 0.0 1703248 246132 pts/2 Sl+ Mar05 18:55 python scripts/munge_fah_data_parallel.py server 20334 1.4 0.0 1690012 232904 pts/2 Sl+ Mar05 19:08 python scripts/munge_fah_data_parallel.py server 20335 1.4 0.0 1690408 233252 pts/2 Sl+ Mar05 19:09 python scripts/munge_fah_data_parallel.py server 20336 1.3 0.0 1709476 252388 pts/2 Sl+ Mar05 18:30 python scripts/munge_fah_data_parallel.py server 20337 1.3 0.0 1712448 255320 pts/2 Sl+ Mar05 18:18 python scripts/munge_fah_data_parallel.py server 20338 1.3 0.0 1704372 247252 pts/2 Sl+ Mar05 18:24 python scripts/munge_fah_data_parallel.py server 20339 1.3 0.0 1699160 242000 pts/2 Sl+ Mar05 18:40 python scripts/munge_fah_data_parallel.py server 20340 1.4 0.0 1702460 245336 pts/2 Sl+ Mar05 19:11 python scripts/munge_fah_data_parallel.py server 20341 1.4 0.0 1701524 244396 pts/2 Sl+ Mar05 20:22 python scripts/munge_fah_data_parallel.py server 26193 1.7 0.0 1883380 229704 pts/2 Sl+ Mar05 23:33 python scripts/munge_fah_data_parallel.py server 26194 1.4 0.0 1812140 158436 pts/2 Sl+ Mar05 19:36 python scripts/munge_fah_data_parallel.py server 26195 1.8 0.0 1869884 216188 pts/2 Sl+ Mar05 25:22 python scripts/munge_fah_data_parallel.py server 26196 1.2 0.0 1810116 156572 pts/2 Sl+ Mar05 16:21 python scripts/munge_fah_data_parallel.py server 26197 1.5 0.0 1877644 223968 pts/2 Sl+ Mar05 20:27 python scripts/munge_fah_data_parallel.py server 26198 1.3 0.0 1890612 236716 pts/2 Sl+ Mar05 18:41 python scripts/munge_fah_data_parallel.py server 26199 1.2 0.0 1832132 178428 pts/2 Sl+ Mar05 16:36 python scripts/munge_fah_data_parallel.py server 26200 1.2 0.0 1834072 180376 pts/2 Sl+ Mar05 16:31 python scripts/munge_fah_data_parallel.py server 26201 1.2 0.0 1836188 182488 pts/2 Sl+ Mar05 16:36 python scripts/munge_fah_data_parallel.py server 26202 1.3 0.0 1837408 183704 pts/2 Sl+ Mar05 18:19 python scripts/munge_fah_data_parallel.py server 26203 1.4 0.0 1837384 183688 pts/2 Sl+ Mar05 18:49 python scripts/munge_fah_data_parallel.py server 26204 1.4 0.0 1885584 231908 pts/2 Sl+ Mar05 19:50 python scripts/munge_fah_data_parallel.py server 26205 1.5 0.0 1834780 181088 pts/2 Sl+ Mar05 21:21 python scripts/munge_fah_data_parallel.py server 26206 1.2 0.0 1837000 183404 pts/2 Sl+ Mar05 16:07 python scripts/munge_fah_data_parallel.py server 26207 1.7 0.0 1870032 216340 pts/2 Sl+ Mar05 23:02 python scripts/munge_fah_data_parallel.py server 26208 1.5 0.0 1840040 186352 pts/2 Sl+ Mar05 21:09 python scripts/munge_fah_data_parallel.py server 27094 1.2 0.7 3846420 1994892 pts/2 Sl+ Mar05 16:33 python scripts/munge_fah_data_parallel.py server 27095 1.2 0.5 3231804 1381336 pts/2 Sl+ Mar05 16:05 python scripts/munge_fah_data_parallel.py server 27096 1.2 0.0 2063120 212792 pts/2 Sl+ Mar05 16:39 python scripts/munge_fah_data_parallel.py server 27097 1.2 0.5 3289964 1439472 pts/2 Sl+ Mar05 16:04 python scripts/munge_fah_data_parallel.py server 27098 1.4 0.1 3845104 301288 pts/2 Sl+ Mar05 18:49 python scripts/munge_fah_data_parallel.py server 27099 1.1 0.0 2073684 223376 pts/2 Sl+ Mar05 15:29 python scripts/munge_fah_data_parallel.py server 27100 1.2 0.0 2064280 213964 pts/2 Sl+ Mar05 16:54 python scripts/munge_fah_data_parallel.py server 27101 1.3 0.0 2102388 252064 pts/2 Sl+ Mar05 18:40 python scripts/munge_fah_data_parallel.py server 27102 1.2 0.0 2089488 238960 pts/2 Sl+ Mar05 16:45 python scripts/munge_fah_data_parallel.py server 27103 1.2 0.0 2031412 181088 pts/2 Sl+ Mar05 16:24 python scripts/munge_fah_data_parallel.py server 27104 1.2 0.3 3624708 1049548 pts/2 Sl+ Mar05 16:26 python scripts/munge_fah_data_parallel.py server 27105 1.2 0.0 2053292 202972 pts/2 Sl+ Mar05 16:55 python scripts/munge_fah_data_parallel.py server 27106 1.0 0.9 4370772 2519744 pts/2 Sl+ Mar05 14:32 python scripts/munge_fah_data_parallel.py server 27107 1.2 0.0 2043076 192916 pts/2 Sl+ Mar05 17:17 python scripts/munge_fah_data_parallel.py server 27108 1.2 0.0 2028504 178220 pts/2 Sl+ Mar05 17:06 python scripts/munge_fah_data_parallel.py server 27109 1.1 0.1 2224816 374408 pts/2 Sl+ Mar05 15:21 python scripts/munge_fah_data_parallel.py server 27223 0.4 0.0 2270008 223160 pts/2 Sl+ Mar05 6:13 python scripts/munge_fah_data_parallel.py server 27224 0.4 0.0 2273880 227032 pts/2 Sl+ Mar05 6:10 python scripts/munge_fah_data_parallel.py server 27225 0.4 0.0 2270412 223280 pts/2 Sl+ Mar05 5:54 python scripts/munge_fah_data_parallel.py server 27226 0.5 0.0 2278600 231748 pts/2 Sl+ Mar05 7:22 python scripts/munge_fah_data_parallel.py server 27227 0.5 0.1 2344860 297824 pts/2 Sl+ Mar05 6:55 python scripts/munge_fah_data_parallel.py server 27228 0.6 0.0 2270788 223940 pts/2 Sl+ Mar05 8:42 python scripts/munge_fah_data_parallel.py server 27229 0.4 0.0 2279592 231936 pts/2 Sl+ Mar05 6:05 python scripts/munge_fah_data_parallel.py server 27230 0.5 0.0 2275256 227600 pts/2 Sl+ Mar05 7:25 python scripts/munge_fah_data_parallel.py server 27231 0.5 0.0 2278296 231196 pts/2 Sl+ Mar05 7:16 python scripts/munge_fah_data_parallel.py server 27232 0.8 0.0 2271308 224460 pts/2 Sl+ Mar05 11:47 python scripts/munge_fah_data_parallel.py server 27233 0.6 0.0 2272084 225232 pts/2 Sl+ Mar05 8:17 python scripts/munge_fah_data_parallel.py server 27234 0.5 0.0 2272752 225648 pts/2 Sl+ Mar05 7:58 python scripts/munge_fah_data_parallel.py server 27235 0.7 0.1 2354816 308028 pts/2 Sl+ Mar05 9:56 python scripts/munge_fah_data_parallel.py server 27236 0.5 0.0 2278892 232044 pts/2 Sl+ Mar05 7:18 python scripts/munge_fah_data_parallel.py server 27237 0.6 0.0 2269560 222712 pts/2 Sl+ Mar05 9:14 python scripts/munge_fah_data_parallel.py server 27238 0.5 0.1 2379544 331956 pts/2 Sl+ Mar05 7:16 python scripts/munge_fah_data_parallel.py server 29398 1.1 0.0 2479236 235556 pts/2 Sl+ Mar05 15:30 python scripts/munge_fah_data_parallel.py server 29399 1.1 0.0 2462796 218812 pts/2 Sl+ Mar05 15:26 python scripts/munge_fah_data_parallel.py server 29400 1.1 0.0 2462768 218996 pts/2 Sl+ Mar05 15:42 python scripts/munge_fah_data_parallel.py server 29401 1.1 0.0 2462936 219124 pts/2 Sl+ Mar05 15:25 python scripts/munge_fah_data_parallel.py server 29402 1.1 0.0 2471908 228204 pts/2 Sl+ Mar05 14:33 python scripts/munge_fah_data_parallel.py server 29403 1.1 0.0 2462984 219132 pts/2 Sl+ Mar05 15:40 python scripts/munge_fah_data_parallel.py server 29404 1.1 0.0 2462424 218460 pts/2 Sl+ Mar05 15:28 python scripts/munge_fah_data_parallel.py server 29405 1.1 0.0 2461688 218128 pts/2 Sl+ Mar05 15:27 python scripts/munge_fah_data_parallel.py server 29406 1.1 0.0 2470880 227200 pts/2 Sl+ Mar05 15:28 python scripts/munge_fah_data_parallel.py server 29407 1.1 0.0 2463736 220040 pts/2 Sl+ Mar05 15:14 python scripts/munge_fah_data_parallel.py server 29408 1.1 0.0 2479196 235512 pts/2 Sl+ Mar05 15:36 python scripts/munge_fah_data_parallel.py server 29409 1.1 0.0 2462608 218700 pts/2 Sl+ Mar05 15:42 python scripts/munge_fah_data_parallel.py server 29410 1.1 0.0 2461208 217660 pts/2 Sl+ Mar05 15:40 python scripts/munge_fah_data_parallel.py server 29411 1.1 0.0 2462808 219008 pts/2 Sl+ Mar05 15:06 python scripts/munge_fah_data_parallel.py server 29412 0.9 0.0 2455472 211964 pts/2 Sl+ Mar05 13:07 python scripts/munge_fah_data_parallel.py server 29413 1.1 0.0 2461120 217576 pts/2 Sl+ Mar05 15:29 python scripts/munge_fah_data_parallel.py server 35491 18.1 0.0 2689844 250080 pts/2 Sl+ Mar05 234:10 python scripts/munge_fah_data_parallel.py server 35492 18.5 0.0 2691048 251248 pts/2 Sl+ Mar05 238:49 python scripts/munge_fah_data_parallel.py server 35493 15.5 0.0 2692812 252816 pts/2 Sl+ Mar05 200:27 python scripts/munge_fah_data_parallel.py server 35494 17.3 0.0 2694780 254796 pts/2 Sl+ Mar05 224:02 python scripts/munge_fah_data_parallel.py server 35495 18.8 0.0 2689688 249920 pts/2 Sl+ Mar05 243:18 python scripts/munge_fah_data_parallel.py server 35496 17.3 0.0 2689296 249544 pts/2 Sl+ Mar05 224:02 python scripts/munge_fah_data_parallel.py server 35497 18.7 0.0 2688980 249332 pts/2 Sl+ Mar05 242:10 python scripts/munge_fah_data_parallel.py server 35498 18.6 0.0 2682528 242760 pts/2 Sl+ Mar05 240:14 python scripts/munge_fah_data_parallel.py server 35499 17.5 0.0 2689508 249748 pts/2 Sl+ Mar05 226:12 python scripts/munge_fah_data_parallel.py server 35500 18.9 0.0 2683200 243436 pts/2 Sl+ Mar05 244:10 python scripts/munge_fah_data_parallel.py server 35501 18.7 0.0 2688964 249208 pts/2 Sl+ Mar05 241:25 python scripts/munge_fah_data_parallel.py server 35502 18.6 0.0 2688764 248996 pts/2 Sl+ Mar05 239:59 python scripts/munge_fah_data_parallel.py server 35503 18.6 0.0 2681480 241820 pts/2 Sl+ Mar05 240:32 python scripts/munge_fah_data_parallel.py server 35504 17.6 0.0 2692360 252596 pts/2 Sl+ Mar05 227:26 python scripts/munge_fah_data_parallel.py server 35505 18.3 0.0 2692180 252412 pts/2 Sl+ Mar05 236:40 python scripts/munge_fah_data_parallel.py server 35506 12.3 0.0 2681512 241744 pts/2 Sl+ Mar05 159:19 python scripts/munge_fah_data_parallel.py server 41492 5.9 0.0 3522420 34276 pts/2 Sl+ Mar05 90:59 python scripts/munge_fah_data_parallel.py server 41506 1.0 0.0 655416 55796 pts/2 Sl+ Mar05 15:50 python scripts/munge_fah_data_parallel.py server 41507 1.2 0.0 658100 111628 pts/2 Sl+ Mar05 18:27 python scripts/munge_fah_data_parallel.py server 41508 1.3 0.0 654448 82388 pts/2 Sl+ Mar05 19:49 python scripts/munge_fah_data_parallel.py server 41509 1.0 0.0 700160 78912 pts/2 Sl+ Mar05 16:04 python scripts/munge_fah_data_parallel.py server 41510 1.0 0.0 654476 100404 pts/2 Sl+ Mar05 16:27 python scripts/munge_fah_data_parallel.py server 41511 1.2 0.0 693644 153608 pts/2 Sl+ Mar05 19:13 python scripts/munge_fah_data_parallel.py server 41512 1.0 0.0 653876 131936 pts/2 Sl+ Mar05 15:36 python scripts/munge_fah_data_parallel.py server 41513 1.2 0.0 690576 149856 pts/2 Sl+ Mar05 18:19 python scripts/munge_fah_data_parallel.py server 41514 1.1 0.0 630932 113380 pts/2 Sl+ Mar05 17:42 python scripts/munge_fah_data_parallel.py server 41515 0.9 0.0 654028 128292 pts/2 Sl+ Mar05 15:00 python scripts/munge_fah_data_parallel.py server 41516 1.1 0.0 653276 124996 pts/2 Sl+ Mar05 17:49 python scripts/munge_fah_data_parallel.py server 41517 1.1 0.0 666628 136676 pts/2 Sl+ Mar05 17:03 python scripts/munge_fah_data_parallel.py server 41518 1.0 0.0 658844 137908 pts/2 Sl+ Mar05 15:59 python scripts/munge_fah_data_parallel.py server 41519 1.1 0.0 632264 113816 pts/2 Sl+ Mar05 17:09 python scripts/munge_fah_data_parallel.py server 41520 1.0 0.0 629700 97100 pts/2 Sl+ Mar05 15:56 python scripts/munge_fah_data_parallel.py server 41521 1.1 0.0 689932 114904 pts/2 Sl+ Mar05 17:55 python scripts/munge_fah_data_parallel.py server 44621 1.2 0.0 869436 127608 pts/2 Sl+ Mar05 18:19 python scripts/munge_fah_data_parallel.py server 44622 1.2 0.0 918464 171636 pts/2 Sl+ Mar05 19:10 python scripts/munge_fah_data_parallel.py server 44623 1.2 0.0 963380 246468 pts/2 Sl+ Mar05 18:26 python scripts/munge_fah_data_parallel.py server 44624 1.2 0.0 861556 135816 pts/2 Sl+ Mar05 18:46 python scripts/munge_fah_data_parallel.py server 44625 1.3 0.0 917684 173152 pts/2 Sl+ Mar05 19:51 python scripts/munge_fah_data_parallel.py server 44626 1.2 0.0 941892 223248 pts/2 Sl+ Mar05 18:19 python scripts/munge_fah_data_parallel.py server 44627 1.2 0.0 910728 208584 pts/2 Sl+ Mar05 18:52 python scripts/munge_fah_data_parallel.py server 44628 1.3 0.0 937112 229232 pts/2 Sl+ Mar05 20:55 python scripts/munge_fah_data_parallel.py server 44629 1.3 0.0 960372 249244 pts/2 Sl+ Mar05 19:53 python scripts/munge_fah_data_parallel.py server 44630 1.2 0.0 861560 161080 pts/2 Sl+ Mar05 18:28 python scripts/munge_fah_data_parallel.py server 44631 1.2 0.0 889292 182700 pts/2 Sl+ Mar05 18:57 python scripts/munge_fah_data_parallel.py server 44632 1.2 0.0 931912 189788 pts/2 Sl+ Mar05 19:16 python scripts/munge_fah_data_parallel.py server 44633 1.2 0.0 911896 171404 pts/2 Sl+ Mar05 18:55 python scripts/munge_fah_data_parallel.py server 44634 1.2 0.0 868900 141580 pts/2 Sl+ Mar05 18:04 python scripts/munge_fah_data_parallel.py server 44635 1.2 0.0 917272 168080 pts/2 Sl+ Mar05 18:50 python scripts/munge_fah_data_parallel.py server 44636 1.2 0.0 925240 221672 pts/2 Sl+ Mar05 19:05 python scripts/munge_fah_data_parallel.py -bash-4.1$ ps xauwww | grep python | wc -l 211
After screen -r and a keyboard terminate:
screen -r
[server@plfah2 FAHMunge]$ ps xauwww | grep python server 3312 0.0 0.0 103308 876 pts/2 S+ 18:43 0:00 grep python root 4077 0.0 0.0 205956 9496 ? S Mar01 0:23 /usr/bin/python -s /usr/sbin/osad --pid-file /var/run/osad.pid
Will investigate. Restarting for now.
Seems relevant: http://stackoverflow.com/questions/30506489/python-multiprocessing-leading-to-many-zombie-processes
Testing a fix.
Seems to be fixed now. I'll have to copy the changes (which I made in situ on plfah2) back to merge them into the repo.
plfah2
Fixed (hopefully!) by #22
It looks like the
multiprocessing
version can keep a lot of processes hanging. After a couple days of running, I noted 211 processes:After
screen -r
and a keyboard terminate:Will investigate. Restarting for now.