search

APS-USAXS / livedata

live data from the APS USAXS instrument
https://usaxslive.xray.aps.anl.gov/
0 stars 0 forks source link

livedata keeps CPU usage at 25-40% #18

Closed prjemian closed 5 years ago

prjemian commented 5 years ago

@jilavsky reports by email:

[livedata is] running all the time with 20-40% cpu.

However, we have 8 cpus and at this time computer is running at base frequency (1.2GHz, max is 3.9GHz), so this whole top may not be actually correct… I suspect it does not account for frequency changes. So it may be nothing…

prjemian commented 5 years ago

Verified that it is, indeed, the livedata task that is using a lot of CPU.

prjemian commented 5 years ago

It's livedata, looking for a file every ten seconds: 2019-05-27 22:56:59.611 (INFO,91329,main,111) /05_19_test.dat does not exist

I restarted livedata, hoping it forgets about that file (during my tests last week).

prjemian commented 5 years ago

That did not fix the problem. The file name comes from a PV that has the file name and a different PV with the directory. The userDir was set to "" and that seems to be part of the problem. Also, the code should accept that the file /05_19_test.dat does not exist and proceed without locking up the CPU. I'll look at fixing that tomorrow.

prjemian commented 5 years ago

Even with correct userDir and specFile PVs, the CPU usage is 40%. So, I've disabled the livedata update and will look at the whole shootin' match tomorrow.

prjemian commented 5 years ago

looking at how long certain steps of the code take to complete, the file transfers to the web server are taking about 2.4 seconds each time they are called (10s interval). That needs to be redesigned. Should be less than 0.1s (my guess).

2019-05-28 09:41:03.101 (INFO,1168,__main__,228) buildReport() completed in 0.064 s
2019-05-28 09:41:03.120 (INFO,1168,__main__,238) writeFile() completed in 0.019 s
2019-05-28 09:41:05.354 (INFO,1168,__main__,271) wwwServerTransfers.scpToWebServer() steps completed in 2.234 s
2019-05-28 09:41:05.355 (INFO,1168,__main__,389) report() completed in 2.318 s
2019-05-28 09:41:13.148 (INFO,1168,__main__,228) buildReport() completed in 0.068 s
2019-05-28 09:41:13.167 (INFO,1168,__main__,238) writeFile() completed in 0.019 s
2019-05-28 09:41:15.477 (INFO,1168,__main__,271) wwwServerTransfers.scpToWebServer() steps completed in 2.310 s
2019-05-28 09:41:15.477 (INFO,1168,__main__,389) report() completed in 2.397 s
2019-05-28 09:41:23.206 (INFO,1168,__main__,228) buildReport() completed in 0.102 s
2019-05-28 09:41:23.235 (INFO,1168,__main__,238) writeFile() completed in 0.029 s
2019-05-28 09:41:25.473 (INFO,1168,__main__,271) wwwServerTransfers.scpToWebServer() steps completed in 2.237 s
2019-05-28 09:41:25.473 (INFO,1168,__main__,389) report() completed in 2.369 s

I'll talk with [our IT group] about how to use NFS instead of scp to move those files.

prjemian commented 5 years ago

With NFS transfer, report() completes in 0.15 - 0.2s.

2019-05-28 10:47:03.174 (INFO,67236,__main__,382) report() completed in 0.213 s
2019-05-28 10:47:03.174 (INFO,67236,__main__,402) checkpoint, 26118 EPICS monitor events received
2019-05-28 10:47:13.157 (INFO,67236,__main__,382) report() completed in 0.148 s
2019-05-28 10:47:23.270 (INFO,67236,__main__,382) report() completed in 0.175 s
2019-05-28 10:47:33.281 (INFO,67236,__main__,382) report() completed in 0.176 s
prjemian commented 5 years ago

Still takes about 25% of one CPU. Not sure that is a problem. Yet.