search

lachlan2k / phatcrack

Modern web-based distributed hashcracking solution, built on hashcat
MIT License
110 stars 8 forks source link

Agent failure on Out of Memory if hashcat process is already dead #33

Closed JSmith-Aura closed 7 months ago

JSmith-Aura commented 8 months ago

Howdy.

Just ran into a condition that causes the agent to die entirely when the graphics card runs out of memory. I believe this is happening due to the process trying to clean up hashcat, but hashcat is already gone leading to a "double close" which triggers and error that kills the agent with here:

https://github.com/lachlan2k/phatcrack/blob/55c45d9e82c288d6038fb4251de8a35075a792c6/agent/internal/hashcat/hashcat.go#L343

Is currently:

func (sess *HashcatSession) Kill() error {
    if sess.proc == nil || sess.proc.Process == nil {
        return nil
    }
    return sess.proc.Process.Kill()
}

Should be: 

func (sess *HashcatSession) Kill() error {
    if sess.proc == nil || sess.proc.Process == nil {
        return nil
    }

        if err := sess.proc.Process.Kill(); err != nil && err != os.ErrProcessDone {
           return err 
        }

    return nil
}

Log for completeness:

Jan 19 14:21:48 node phatcrack-agent[3816646]: 2024/01/19 14:21:48 Running hashcat command: "/opt/crypt/phatcrack-agent/hashcat/hashcat.bin --quiet --session sess-omitted>
Jan 19 14:21:48 node phatcrack-agent[3816646]: 2024/01/19 14:21:48 read stderr: "cuCtxCreate(): out of memory"
Jan 19 14:21:48 node phatcrack-agent[3816646]: 2024/01/19 14:21:48 read stderr: ""
Jan 19 14:21:48 node phatcrack-agent[3816646]: 2024/01/19 14:21:48 read stderr: "cuCtxCreate(): out of memory"
Jan 19 14:21:48 node phatcrack-agent[3816646]: 2024/01/19 14:21:48 read stderr: ""
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: "cuCtxCreate(): out of memory"
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: ""
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: "cuCtxCreate(): out of memory"
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: ""
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: "clCreateContext(): CL_OUT_OF_HOST_MEMORY"
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: ""
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: "clCreateContext(): CL_OUT_OF_HOST_MEMORY"
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: ""
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: "clCreateContext(): CL_OUT_OF_HOST_MEMORY"
Jan 19 14:21:49 node phatcrack-agent[3816646]: 2024/01/19 14:21:49 read stderr: ""
Jan 19 14:21:50 node phatcrack-agent[3816646]: 2024/01/19 14:21:50 read stderr: "clCreateContext(): CL_OUT_OF_HOST_MEMORY"
Jan 19 14:21:50 node phatcrack-agent[3816646]: 2024/01/19 14:21:50 read stderr: ""
Jan 19 14:21:50 node phatcrack-agent[3816646]: 2024/01/19 14:21:50 read stderr: "No devices found/left."
Jan 19 14:21:50 node phatcrack-agent[3816646]: 2024/01/19 14:21:50 read stderr: ""
Jan 19 14:21:50 node phatcrack-agent[3816646]: 2024/01/19 14:21:50 Received: JobKill
Jan 19 14:21:50 node phatcrack-agent[3816646]: 2024/01/19 14:21:50 unrecoverable handler error: error when handling message: os: process already finished
Jan 19 14:21:50 node systemd[1]: phatcrack-agent.service: Main process exited, code=exited, status=1/FAILURE
Jan 19 14:21:51 node systemd[1]: phatcrack-agent.service: Failed with result 'exit-code'.
JSmith-Aura commented 8 months ago

It may also be worth making a failure to kill a process not a fatal error. As half the time hashcat dies on its own anyway.

lachlan2k commented 8 months ago

Thanks for the report and investigation.

Agent error handling is definitely very fragile.

Feel free to PR/contribute your suggested fix as it looks good imo!