blumzi / LAST_issues

A place to discuss and manage LAST issues
0 stars 0 forks source link

M4 and M5 got slow, produced giant log files, which filled up the root directory on LAST0 #24

Open noralinn opened 5 months ago

noralinn commented 5 months ago

Both M5, and later M4, were slow. Restarting nomachine seemed to help at first.

But a bit later we could connect via nomachine, but couldn't open any programs/click anywhere (M5). On M4 Yarin had started observations and then lost the nomachine connection to the computer. Using ssh we saw at the beginning that images were still being saved and just left the mount observing. When checking later, the mount had stopped saving images a bit later.

About an hour later, I got a warning on last0 that the root directory was full and because of that I couldn't use nomachine anymore. When checking, I found 400 GB of log files. /var/log/remote/last04e/last-messages.log had a size of 110 GB and /var/log/remote/last05e/last-messages.log 75 GB.

Eran deleted those files on last0, last04e, and last05e. We restarted the computers of M4 and M5. LAST0 also had to be restarted, because nomachine was still not working. I couldn't shut it down (sudo /usr/NX/bin/nxserver --restart timed out). After restarting everything worked.