Yesterday, we had an issue on our server that coli-ana spawned tons of vc_day_cli2 child processes which didn't terminate, causing the CPU to be at 100% and therefore blocking the whole server. This definitely shouldn't happen. Unfortunately, I wasn't able to find out what the actual issue was. I suspect there might have been one request to the backend which caused some kind of loop or something, but I don't think I can find out what it was now. This should be investigated though.
As temporary measures, I will:
[x] add a timeout to the call, therefore making sure that the child process terminates no matter what
[x] add a debug output that logs notations for which the backend request failed or was terminated
Yesterday, we had an issue on our server that coli-ana spawned tons of
vc_day_cli2child processes which didn't terminate, causing the CPU to be at 100% and therefore blocking the whole server. This definitely shouldn't happen. Unfortunately, I wasn't able to find out what the actual issue was. I suspect there might have been one request to the backend which caused some kind of loop or something, but I don't think I can find out what it was now. This should be investigated though.As temporary measures, I will: