Closed falkamelung closed 2 years ago
Summarizing here a few top activity screens
top
--num_worker 20
S
sjobs JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 8901121 skx-dev idv27073 tg851601 R 1:53:09 1 c506-034 8901268 skx-normal run_02_m tg851601 R 1:14:08 1 c476-001 8901297 skx-normal run_02_m tg851601 R 1:03:59 1 c479-023 8901300 skx-normal run_02_m tg851601 R 1:03:59 1 c479-053 8901343 skx-normal run_02_m tg851601 R 52:44 1 c493-074 8901346 skx-normal run_02_m tg851601 R 52:44 1 c494-053 8901341 skx-normal run_02_m tg851601 R 53:55 1 c493-052 8901360 skx-normal run_02_m tg851601 R 47:56 1 c493-032 8901362 skx-normal run_02_m tg851601 R 47:56 1 c497-051 8901452 skx-normal run_02_m tg851601 R 33:48 1 c506-081 8901444 skx-normal run_02_m tg851601 R 36:14 1 c506-071 8901446 skx-normal run_02_m tg851601 R 36:14 1 c506-072 8901448 skx-normal run_02_m tg851601 R 36:14 1 c506-073 8901450 skx-normal run_02_m tg851601 R 36:14 1 c506-074 8901465 skx-normal run_02_m tg851601 R 30:12 1 c506-091 8901467 skx-normal run_02_m tg851601 R 30:12 1 c506-092 8901469 skx-normal run_02_m tg851601 R 30:12 1 c506-093 8901494 skx-normal run_02_m tg851601 R 19:28 1 c506-083 8901496 skx-normal run_02_m tg851601 R 19:28 1 c506-084 8901528 skx-normal run_02_m tg851601 R 3:04 1 c506-111 8901530 skx-normal run_02_m tg851601 R 3:04 1 c506-112```
top - 23:52:43 up 119 days, 12:22, 7 users, load average: 11.00, 10.94, 13.70 Tasks: 1122 total, 12 running, 1110 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.5 us, 0.0 sy, 0.0 ni, 88.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 187.3 total, 172.8 free, 5.0 used, 9.6 buff/cache GiB Swap: 0.0 total, 0.0 free, 0.0 used. 177.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 330511 tg851601 20 0 663864 198876 5388 R 99.7 0.1 25:08.86 python3 330512 tg851601 20 0 663632 199260 5264 R 99.7 0.1 25:09.08 python3 330649 tg851601 20 0 664088 197676 5388 R 99.7 0.1 25:09.14 python3 330918 tg851601 20 0 664020 197008 5388 R 99.7 0.1 25:09.11 python3 330919 tg851601 20 0 663768 196996 5388 R 99.7 0.1 25:09.11 python3 330920 tg851601 20 0 663812 197096 5388 R 99.7 0.1 25:09.44 python3 330509 tg851601 20 0 663868 199676 5388 R 99.3 0.1 25:08.83 python3 330510 tg851601 20 0 663956 199192 5388 R 99.3 0.1 25:09.00 python3 330537 tg851601 20 0 663776 199616 5388 R 99.3 0.1 25:09.03 python3 330538 tg851601 20 0 663800 199524 5388 R 99.3 0.1 25:08.87 python3 330540 tg851601 20 0 664052 199292 5388 R 99.3 0.1 25:09.46 python3 332399 tg851601 20 0 152912 3508 1656 R 0.7 0.0 0:00.15 top 329156 tg851601 20 0 113192 1476 1196 S 0.0 0.0 0:00.00 slurm_script 329158 tg851601 20 0 113332 1628 1304 S 0.0 0.0 0:00.00 paramrun 329168 tg851601 20 0 113332 856 464 S 0.0 0.0 0:00.00 paramrun 329176 tg851601 20 0 150788 5428 4212 S 0.0 0.0 0:00.07 ssh 329181 tg851601 20 0 177112 4416 1048 S 0.0 0.0 0:00.00 sshd 329182 tg851601 20 0 113320 1788 1456 S 0.0 0.0 0:00.02 bash 329260 tg851601 20 0 113192 1560 1268 S 0.0 0.0 0:00.00 init_launcher 329264 tg851601 20 0 113328 1616 1300 S 0.0 0.0 0:00.00 launcher 329265 tg851601 20 0 113328 1616 1300 S 0.0 0.0 0:00.00 launcher 329281 tg851601 20 0 113328 1616 1300 S 0.0 0.0 0:00.00 launcher 329283 tg851601 20 0 113328 1616 1300 S 0.0 0.0 0:00.00 launcher 330168 tg851601 20 0 620112 112092 37416 S 0.0 0.1 0:01.27 python3 330186 tg851601 20 0 620112 112452 37412 S 0.0 0.1 0:01.24 python3 330325 tg851601 20 0 620116 112620 37416 S 0.0 0.1 0:01.20 python3 330541 tg851601 20 0 417348 75664 1072 S 0.0 0.0 0:00.00 python3 330653 tg851601 20 0 417352 76232 1100 S 0.0 0.0 0:00.00 python3 330657 tg851601 20 0 417352 76196 1068 S 0.0 0.0 0:00.00 python3 330664 tg851601 20 0 417352 76200 1068 S 0.0 0.0 0:00.00 python3 330703 tg851601 20 0 620116 112828 37412 S 0.0 0.1 0:01.18 python3 330921 tg851601 20 0 417352 76424 1088 S 0.0 0.0 0:00.00 python3 332265 tg851601 20 0 177288 4672 1188 S 0.0 0.0 0:00.00 sshd 332266 tg851601 20 0 115832 2612 1768 S 0.0 0.0 0:00.03 bash `` A younder job runs fine:
top - 23:57:11 up 286 days, 15:13, 1 user, load average: 44.84, 44.96, 43.14 Tasks: 1150 total, 43 running, 1107 sleeping, 0 stopped, 0 zombie %Cpu(s): 45.0 us, 0.1 sy, 0.0 ni, 54.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 187.3 total, 161.2 free, 9.5 used, 16.6 buff/cache GiB Swap: 0.0 total, 0.0 free, 0.0 used. 172.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 365959 tg851601 20 0 663804 197608 5388 R 100.0 0.1 47:08.00 python3 365971 tg851601 20 0 664324 197880 5268 R 100.0 0.1 47:07.95 python3 365983 tg851601 20 0 663544 199060 5264 R 100.0 0.1 47:07.96 python3 365991 tg851601 20 0 664324 199924 5272 R 100.0 0.1 47:07.94 python3 365935 tg851601 20 0 664324 197804 5272 R 99.7 0.1 47:08.04 python3 365936 tg851601 20 0 663648 197412 5268 R 99.7 0.1 47:08.01 python3 365937 tg851601 20 0 664324 198200 5272 R 99.7 0.1 47:08.14 python3 365938 tg851601 20 0 664324 197812 5272 R 99.7 0.1 47:07.94 python3 365942 tg851601 20 0 664324 198428 5392 R 99.7 0.1 47:08.18 python3 365944 tg851601 20 0 664324 198240 5392 R 99.7 0.1 47:08.15 python3 365945 tg851601 20 0 663828 197840 5388 R 99.7 0.1 47:08.16 python3 365949 tg851601 20 0 663792 197204 5268 R 99.7 0.1 47:08.01 python3 365950 tg851601 20 0 664324 198412 5392 R 99.7 0.1 47:07.79 python3 365951 tg851601 20 0 663804 197376 5388 R 99.7 0.1 47:07.88 python3 365952 tg851601 20 0 664324 197844 5268 R 99.7 0.1 47:07.99 python3 365953 tg851601 20 0 664324 198088 5392 R 99.7 0.1 47:08.22 python3 365964 tg851601 20 0 664324 198016 5392 R 99.7 0.1 47:07.93 python3 365966 tg851601 20 0 663668 197016 5268 R 99.7 0.1 47:07.45 python3 365967 tg851601 20 0 658576 192268 5388 R 99.7 0.1 47:07.92 python3 365968 tg851601 20 0 664324 197804 5272 R 99.7 0.1 47:07.85 python3 365970 tg851601 20 0 663744 197224 5268 R 99.7 0.1 47:07.99 python3 365972 tg851601 20 0 663844 199232 5264 R 99.7 0.1 47:07.99 python3 365977 tg851601 20 0 664324 197892 5272 R 99.7 0.1 47:07.87 python3 365978 tg851601 20 0 664324 199848 5268 R 99.7 0.1 47:07.78 python3 365984 tg851601 20 0 664324 200108 5392 R 99.7 0.1 47:08.07 python3 365986 tg851601 20 0 664324 199852 5272 R 99.7 0.1 47:08.35 python3 365993 tg851601 20 0 663816 197392 5388 R 99.7 0.1 47:07.56 python3 365995 tg851601 20 0 664324 199868 5272 R 99.7 0.1 47:08.02 python3 365996 tg851601 20 0 664324 199940 5268 R 99.7 0.1 47:07.87 python3 365997 tg851601 20 0 664324 199868 5268 R 99.7 0.1 47:08.05 python3 365998 tg851601 20 0 664324 198004 5392 R 99.7 0.1 47:07.60 python3 365999 tg851601 20 0 664324 199868 5268 R 99.7 0.1 47:08.12 python3 366000 tg851601 20 0 664324 199924 5272 R 99.7 0.1 47:07.65 python3 366001 tg851601 20 0 663740 197296 5388 R 99.7 0.1 47:07.59 python3 366005 tg851601 20 0 664324 199896 5268 R 99.7 0.1 47:08.06 python3 366022 tg851601 20 0 664324 198056 5268 R 99.7 0.1 47:07.94 python3 366023 tg851601 20 0 663668 197124 5264 R 99.7 0.1 47:07.63 python3 366024 tg851601 20 0 664324 198224 5392 R 99.7 0.1 47:08.01 python3 366025 tg851601 20 0 663544 197268 5268 R 99.7 0.1 47:08.03 python3 366051 tg851601 20 0 663960 197388 5268 R 99.7 0.1 47:08.08 python3 366052 tg851601 20 0 664324 197892 5272 R 99.7 0.1 47:07.87 python3 366053 tg851601 20 0 663864 197300 5268 R 99.7 0.1 47:08.08 python3 365957 tg851601 20 0 550484 80772 5524 S 83.8 0.0 47:07.54 python3 365958 tg851601 20 0 551408 81564 5540 S 27.1 0.0 47:05.78 python3 365943 tg851601 20 0 549892 80984 5536 S 11.9 0.0 47:05.30 python3 368530 tg851601 20 0 153056 3548 1656 R 0.7 0.0 0:00.19 top 364569 tg851601 20 0 113192 1476 1196 S 0.0 0.0 0:00.00 slurm_script 364571 tg851601 20 0 113332 1628 1304 S 0.0 0.0 0:00.00 paramrun 364581 tg851601 20 0 113332 848 456 S 0.0 0.0 0:00.00 paramrun 364589 tg851601 20 0 150788 5432 4212 S 0.0 0.0 0:00.06 ssh 364593 tg851601 20 0 177476 4472 1044 S 0.0 0.0 0:00.00 sshd
top - 00:02:44 up 74 days, 13:21, 5 users, load average: 1.00, 1.01, 3.71 Tasks: 1107 total, 2 running, 1105 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 187.3 total, 181.6 free, 3.3 used, 2.4 buff/cache GiB Swap: 0.0 total, 0.0 free, 0.0 used. 181.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 104481 tg851601 20 0 663932 199736 5388 R 99.3 0.1 34:59.97 python3 106537 tg851601 20 0 152912 3492 1656 R 0.3 0.0 0:00.11 top 102925 tg851601 20 0 113192 1476 1192 S 0.0 0.0 0:00.00 slurm_script 102927 tg851601 20 0 113332 1628 1304 S 0.0 0.0 0:00.00 paramrun 102937 tg851601 20 0 113332 852 460 S 0.0 0.0 0:00.00 paramrun 102945 tg851601 20 0 150788 5428 4212 S 0.0 0.0 0:00.07 ssh 102948 tg851601 20 0 177112 4420 1052 S 0.0 0.0 0:00.00 sshd 102949 tg851601 20 0 113320 1784 1456 S 0.0 0.0 0:00.01 bash 103026 tg851601 20 0 113192 1560 1268 S 0.0 0.0 0:00.00 init_launcher 103047 tg851601 20 0 113328 1616 1300 S 0.0 0.0 0:00.00 launcher 104232 tg851601 20 0 620112 112280 37412 S 0.0 0.1 0:01.19 python3 104484 tg851601 20 0 417348 75880 1092 S 0.0 0.0 0:00.00 python3 104485 tg851601 20 0 417348 75888 1092 S 0.0 0.0 0:00.00 python3 104488 tg851601 20 0 417348 75884 1092 S 0.0 0.0 0:00.00 python3 106404 tg851601 20 0 177104 4428 1064 S 0.0 0.0 0:00.00 sshd 106405 tg851601 20 0 115836 2612 1768 S 0.0 0.0 0:00.03 bash
Summarizing here a few
topactivity screens--num_worker 20, about half of workers showStop - 23:57:11 up 286 days, 15:13, 1 user, load average: 44.84, 44.96, 43.14 Tasks: 1150 total, 43 running, 1107 sleeping, 0 stopped, 0 zombie %Cpu(s): 45.0 us, 0.1 sy, 0.0 ni, 54.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 187.3 total, 161.2 free, 9.5 used, 16.6 buff/cache GiB Swap: 0.0 total, 0.0 free, 0.0 used. 172.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
365959 tg851601 20 0 663804 197608 5388 R 100.0 0.1 47:08.00 python3
365971 tg851601 20 0 664324 197880 5268 R 100.0 0.1 47:07.95 python3
365983 tg851601 20 0 663544 199060 5264 R 100.0 0.1 47:07.96 python3
365991 tg851601 20 0 664324 199924 5272 R 100.0 0.1 47:07.94 python3
365935 tg851601 20 0 664324 197804 5272 R 99.7 0.1 47:08.04 python3
365936 tg851601 20 0 663648 197412 5268 R 99.7 0.1 47:08.01 python3
365937 tg851601 20 0 664324 198200 5272 R 99.7 0.1 47:08.14 python3
365938 tg851601 20 0 664324 197812 5272 R 99.7 0.1 47:07.94 python3
365942 tg851601 20 0 664324 198428 5392 R 99.7 0.1 47:08.18 python3
365944 tg851601 20 0 664324 198240 5392 R 99.7 0.1 47:08.15 python3
365945 tg851601 20 0 663828 197840 5388 R 99.7 0.1 47:08.16 python3
365949 tg851601 20 0 663792 197204 5268 R 99.7 0.1 47:08.01 python3
365950 tg851601 20 0 664324 198412 5392 R 99.7 0.1 47:07.79 python3
365951 tg851601 20 0 663804 197376 5388 R 99.7 0.1 47:07.88 python3
365952 tg851601 20 0 664324 197844 5268 R 99.7 0.1 47:07.99 python3
365953 tg851601 20 0 664324 198088 5392 R 99.7 0.1 47:08.22 python3
365964 tg851601 20 0 664324 198016 5392 R 99.7 0.1 47:07.93 python3
365966 tg851601 20 0 663668 197016 5268 R 99.7 0.1 47:07.45 python3
365967 tg851601 20 0 658576 192268 5388 R 99.7 0.1 47:07.92 python3
365968 tg851601 20 0 664324 197804 5272 R 99.7 0.1 47:07.85 python3
365970 tg851601 20 0 663744 197224 5268 R 99.7 0.1 47:07.99 python3
365972 tg851601 20 0 663844 199232 5264 R 99.7 0.1 47:07.99 python3
365977 tg851601 20 0 664324 197892 5272 R 99.7 0.1 47:07.87 python3
365978 tg851601 20 0 664324 199848 5268 R 99.7 0.1 47:07.78 python3
365984 tg851601 20 0 664324 200108 5392 R 99.7 0.1 47:08.07 python3
365986 tg851601 20 0 664324 199852 5272 R 99.7 0.1 47:08.35 python3
365993 tg851601 20 0 663816 197392 5388 R 99.7 0.1 47:07.56 python3
365995 tg851601 20 0 664324 199868 5272 R 99.7 0.1 47:08.02 python3
365996 tg851601 20 0 664324 199940 5268 R 99.7 0.1 47:07.87 python3
365997 tg851601 20 0 664324 199868 5268 R 99.7 0.1 47:08.05 python3
365998 tg851601 20 0 664324 198004 5392 R 99.7 0.1 47:07.60 python3
365999 tg851601 20 0 664324 199868 5268 R 99.7 0.1 47:08.12 python3
366000 tg851601 20 0 664324 199924 5272 R 99.7 0.1 47:07.65 python3
366001 tg851601 20 0 663740 197296 5388 R 99.7 0.1 47:07.59 python3
366005 tg851601 20 0 664324 199896 5268 R 99.7 0.1 47:08.06 python3
366022 tg851601 20 0 664324 198056 5268 R 99.7 0.1 47:07.94 python3
366023 tg851601 20 0 663668 197124 5264 R 99.7 0.1 47:07.63 python3
366024 tg851601 20 0 664324 198224 5392 R 99.7 0.1 47:08.01 python3
366025 tg851601 20 0 663544 197268 5268 R 99.7 0.1 47:08.03 python3
366051 tg851601 20 0 663960 197388 5268 R 99.7 0.1 47:08.08 python3
366052 tg851601 20 0 664324 197892 5272 R 99.7 0.1 47:07.87 python3
366053 tg851601 20 0 663864 197300 5268 R 99.7 0.1 47:08.08 python3
365957 tg851601 20 0 550484 80772 5524 S 83.8 0.0 47:07.54 python3
365958 tg851601 20 0 551408 81564 5540 S 27.1 0.0 47:05.78 python3
365943 tg851601 20 0 549892 80984 5536 S 11.9 0.0 47:05.30 python3
368530 tg851601 20 0 153056 3548 1656 R 0.7 0.0 0:00.19 top
364569 tg851601 20 0 113192 1476 1196 S 0.0 0.0 0:00.00 slurm_script
364571 tg851601 20 0 113332 1628 1304 S 0.0 0.0 0:00.00 paramrun
364581 tg851601 20 0 113332 848 456 S 0.0 0.0 0:00.00 paramrun
364589 tg851601 20 0 150788 5432 4212 S 0.0 0.0 0:00.06 ssh
364593 tg851601 20 0 177476 4472 1044 S 0.0 0.0 0:00.00 sshd
top - 00:02:44 up 74 days, 13:21, 5 users, load average: 1.00, 1.01, 3.71 Tasks: 1107 total, 2 running, 1105 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st GiB Mem : 187.3 total, 181.6 free, 3.3 used, 2.4 buff/cache GiB Swap: 0.0 total, 0.0 free, 0.0 used. 181.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
104481 tg851601 20 0 663932 199736 5388 R 99.3 0.1 34:59.97 python3
106537 tg851601 20 0 152912 3492 1656 R 0.3 0.0 0:00.11 top
102925 tg851601 20 0 113192 1476 1192 S 0.0 0.0 0:00.00 slurm_script
102927 tg851601 20 0 113332 1628 1304 S 0.0 0.0 0:00.00 paramrun
102937 tg851601 20 0 113332 852 460 S 0.0 0.0 0:00.00 paramrun
102945 tg851601 20 0 150788 5428 4212 S 0.0 0.0 0:00.07 ssh
102948 tg851601 20 0 177112 4420 1052 S 0.0 0.0 0:00.00 sshd
102949 tg851601 20 0 113320 1784 1456 S 0.0 0.0 0:00.01 bash
103026 tg851601 20 0 113192 1560 1268 S 0.0 0.0 0:00.00 init_launcher
103047 tg851601 20 0 113328 1616 1300 S 0.0 0.0 0:00.00 launcher
104232 tg851601 20 0 620112 112280 37412 S 0.0 0.1 0:01.19 python3
104484 tg851601 20 0 417348 75880 1092 S 0.0 0.0 0:00.00 python3
104485 tg851601 20 0 417348 75888 1092 S 0.0 0.0 0:00.00 python3
104488 tg851601 20 0 417348 75884 1092 S 0.0 0.0 0:00.00 python3
106404 tg851601 20 0 177104 4428 1064 S 0.0 0.0 0:00.00 sshd
106405 tg851601 20 0 115836 2612 1768 S 0.0 0.0 0:00.03 bash