search

APS-USAXS / livedata

live data from the APS USAXS instrument
https://usaxslive.xray.aps.anl.gov/
0 stars 0 forks source link

2 python processes using 100% CPU - Why #50

Closed prjemian closed 2 years ago

prjemian commented 3 years ago 
top - 20:44:28 up 119 days, 20:23, 20 users,  load average: 2.38, 2.43, 2.47
Tasks: 664 total,   6 running, 657 sleeping,   0 stopped,   1 zombie
%Cpu(s): 25.5 us, 17.0 sy,  0.0 ni, 57.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 15974900 total,   277940 free,  6147000 used,  9549960 buff/cache
KiB Swap: 33554428 total, 32197628 free,  1356800 used.  9393772 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                 
45851 usaxs     20   0  770160  87784  16888 R 106.2  0.5  11888:59 python                                                                  
52492 nx        20   0 1433012 989.9m   3796 R 106.2  6.3  10765:52 nxserver.bin                                                            
13654 usaxs     20   0  793768 111464  16888 R 100.0  0.7  14118:22 python
prjemian commented 3 years ago

The number on the left column of the top report is the process identifier (PID). Add this to the ps command to get more info about that process. There are lots of options to get even more details. Just the simple one here:

bash-4.2$ ps 13654
  PID TTY      STAT   TIME COMMAND
13654 ?        Rl   14119:09 /APSshare/anaconda/x86_64/bin/python /home/beams/USAXS/Documents/eclipse/USAXS/livedata/specplotsAllScans.py /s
bash-4.2$ ps 45851
  PID TTY      STAT   TIME COMMAND
45851 ?        Rl   11889:48 /APSshare/anaconda/x86_64/bin/python /home/beams/USAXS/Documents/eclipse/USAXS/livedata/specplotsAllScans.py /s

Two instances of the same program, specplotsAllScans.py. Sounds buggy. Both can be killed.

prjemian commented 3 years ago

Killed both, look again at top:

bash-4.2$ kill -9 13654 45851
bash-4.2$ top | head -10

top - 20:48:54 up 119 days, 20:28, 20 users,  load average: 1.87, 2.32, 2.44
Tasks: 661 total,   1 running, 659 sleeping,   0 stopped,   1 zombie
%Cpu(s):  6.0 us,  2.6 sy,  0.0 ni, 91.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 15974900 total,   438664 free,  5984304 used,  9551932 buff/cache
KiB Swap: 33554428 total, 32197628 free,  1356800 used.  9556536 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                 
25385 usaxs     20   0 1723944 243848   8344 S  25.0  1.5  33274:51 python                                                                  
11779 usaxs     20   0  160784   2916   1808 R  12.5  0.0   0:00.05 top                                                                     
  377 root      20   0   52380   1332   1216 S   6.2  0.0   2606:17 plymouthd
prjemian commented 3 years ago

Good. No more of that program now. This one is pvwatch.py, the livedata page main program.

bash-4.2$ ps 25385
  PID TTY      STAT   TIME COMMAND
25385 ?        Sl   33275:08 /APSshare/anaconda/x86_64/bin/python /home/beams/USAXS/Documents/eclipse/USAXS/livedata/pvwatch.py
prjemian commented 3 years ago

Something wrong with specplotsAllScans.py, so leaving this issue open as a BUG report now.

prjemian commented 2 years ago

The CPU usage can be improved by reducing the number of times a plot is regenerated, as described in https://github.com/APS-USAXS/livedata/issues/55#issuecomment-938326458. It will be necessary to compare the time stamp of the last plot available with the time the data file was last modified. Might need to remake that last plot if there is additional data in its scan. Then, make new plots for any new scans.

prjemian commented 2 years ago

To stop the buildSpecPlots.sh script from starting multiple processes, create a lock file while processing is running, then delete it afterwards. Before running the process, verify that the lock file does not already exist.

This can fail if the shell script exits with an error before it has a chance to delete the lock file. Is there a better way?

prjemian commented 2 years ago

optimization: for each SPEC data file, remake the last plot (if any) and make plots for any new scans