tgross
commented
1 month ago Hi @guifran001! I can't really figure out why your postgres process exited, but it looks like the shutdown of the executor is the issue here. When the compile-keycloak pre-start task finishes, we see the following logs:
2024-07-19T17:49:23.430Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=876ad764-d20b-600a-adcb-8f22186401f7 task=compile-keycloak type=Terminated msg="Exit Code: 0" failed=false 2024-07-19T17:49:23.438Z [ERROR] client.driver_mgr.raw_exec: destroying executor failed: driver=raw_exec error="rpc error: code = Unknown desc = executor failed to find process: OpenProcess: The parameter is incorrect." 2024-07-19T17:49:23.464Z [INFO] client.driver_mgr.raw_exec.executor: plugin process exited: alloc_id=876ad764-d20b-600a-adcb-8f22186401f7 driver=raw_exec task_name=compile-keycloak plugin=C:\install\nomad-1.8.2\bin\nomad.exe id=4688
This error is coming from executor.go#L612. To hit this code at all, the executor still has the process PID it's kept around in memory, but when we call os.FindProcess that hits this code in the Go stdlib, which is just a Windows syscall:
func findProcess(pid int) (p *Process, err error) {
const da = syscall.STANDARD_RIGHTS_READ |
syscall.PROCESS_QUERY_INFORMATION | syscall.SYNCHRONIZE
h, e := syscall.OpenProcess(da, false, uint32(pid))
if e != nil {
return nil, NewSyscallError("OpenProcess", e)
}
return newProcess(pid, uintptr(h)), nil
}Is there any possibility that your task daemonizes? That is, does it double fork and then closes the first forking process to divorce the child from the grandparent (the Nomad executor in this case)? If not, then I'm not sure what's up... you're running Nomad as a privileged user, right?
Could it be related to #23595 ?
That's an error trying to access Linux cgroups, so highly unlikely to be related.
guifran001
PayStreamArmitsteadJ
ORuessel
Nomad version
Nomad v1.8.2 BuildDate 2024-07-16T08:50:09Z Revision 7f0822c1e4f25907d9f60e2d595411950dd1bd28
Operating system and Environment details
Windows driver: raw_exec
Issue
Jobs are stopping unexpectedly in nomad, but the processes are still alive in Windows, so when nomad tries to restart them, it fails because the port of application is already bound or a lock file is already lock.
It might not be 100% reproductible as we have an automated test that deploys a nomad instance and starts some jobs that never failed in 1.7.3, but now is failing since our update to 1.8.2. The test has been successful once with 1.8.2.
The automated test deploy consul 1.17.2 as both server and client, then nomad 1.8.2 as both server and client, then starts a postgres job and a keycloak job.
From the logs, we can see that postgres and keycloak start, then postgres stops and keycloak stops with Exit Code 2. Then they are stuck in a restart loop as the orignial postgres and keycloak processes are still executing.
From the log, we can see the Exit Code 2 that is usually reserved to file not found error. Could it be related to #23595 ?
Reproduction steps
80%. The test is using pulumi to run the jobs. Each job is run sequentially
Expected Result
postgres and keycloak run fine
Actual Result
postgres and keycloak are stuck in a restart loop while a detached process of postgres and one from keycloak started by nomad are still running while nomad try to start new ones.
Job file (if appropriate)
Will be provided if needed.
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)