collection stopping after autoupdate

[brendam]

I've found some of our dmi-tcat servers have stopped collecting. Looking into it, the processes are not running but the files are present in the proc directory stopping them being restarted. Looking at the logs it seems that it might be happening during an auto update:

2019-08-01 03:56:23     not yet executing auto-update, because the last commit is less than a day ol
d
2019-08-01 03:56:23     script track is running with pid [3926] and has been idle for 31 seconds
2019-08-01 03:58:01     controller.php already running, skipping this check
2019-08-01 03:59:01     controller.php already running, skipping this check

in the proc directory are these files:

-rw-rw-r-- 1 tcat tcat 15 Aug  2 02:00 track.procinfo
-rw-rw-r-- 1 tcat tcat 51 Aug  1 03:57 controller.lock
-rw-rw-r-- 1 tcat tcat 46 Aug  1 03:33 track.lock
-rw-rw-r-- 1 tcat tcat 48 Jul  4 02:20 upgrade.lock

top -u tcat gives:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 4307 tcat      20   0    4628    816    748 S  0.0  0.0   0:00.00 sh
 4308 tcat      20   0  315428  26284  19168 S  0.0  1.3   2:10.60 php

There are track.log or follow.log files after the 1st, but they are 0 bytes.

The problem has happened on both track and follow servers. We have been having some intermittent networking issues with our servers, so that might be part of the problem.

Is there any other information that would be useful?

[brendam]

Just checked, and the servers are up to date with origin/master.

[ErikBorra]

Closing this issue as it has gone stale. Feel free to reopen.