Homebrew / homebrew-services

🚀 Manage background services using the daemon manager launchctl on macOS or systemctl on Linux.
BSD 2-Clause "Simplified" License
1.99k stars 206 forks source link

PID files aren't getting cleaned up after a hard restart/crash #727

Open GottemHams opened 2 weeks ago

GottemHams commented 2 weeks ago

The crash was not related to anything Homebrew, but after it booted back up I noticed Apache wouldn't start. Trying some things, it looks like the PID file it originally created keeps lingering around and Apache itself refuses to start because of it. The same will probably true for any other service that uses PID files on its own.

Some shell output, with extra line breaks to make it a bit more readable:

$ brew services start httpd
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/gottemhams/Library/LaunchAgents/homebrew.mxcl.httpd.plist` exited with 5.

$ brew services restart httpd
Stopping `httpd`... (might take a while)
==> Successfully stopped `httpd` (label: homebrew.mxcl.httpd)
==> Successfully started `httpd` (label: homebrew.mxcl.httpd)

$ brew services | grep ^httpd
httpd   stopped         gottemhams ~/Library/LaunchAgents/homebrew.mxcl.httpd.plist

$ /opt/homebrew/opt/httpd/bin/httpd -D FOREGROUND
httpd (pid 2490) already running

$ echo $?
0

$ ps -p 2490
  PID TTY           TIME CMD
 2490 ??         0:00.25 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid C128B

$ cat /opt/homebrew/var/run/httpd/httpd.pid
2490

$ rm /opt/homebrew/var/run/httpd/httpd.pid
$ brew services start httpd
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/gottemhams/Library/LaunchAgents/homebrew.mxcl.httpd.plist` exited with 5.

$ brew services stop httpd
Stopping `httpd`... (might take a while)
==> Successfully stopped `httpd` (label: homebrew.mxcl.httpd)

$ cat /opt/homebrew/var/run/httpd/httpd.pid 
cat: /opt/homebrew/var/run/httpd/httpd.pid: No such file or directory

$ brew services start httpd
==> Successfully started `httpd` (label: homebrew.mxcl.httpd)

$ cat /opt/homebrew/var/run/httpd/httpd.pid 
98579

$ ps -p 98579
  PID TTY           TIME CMD
98579 ??         0:00.12 /opt/homebrew/opt/httpd/bin/httpd -D FOREGROUND

Since Homebrew does know that the service is not actually running despite there being an actual process with that exact PID, I would presume that it's fairly easy to recognise if the PID file should be removed/emptied upon receiving a start command?

Also notice how I need to explicitly stop the service first, even though brew services already says stopped. I think Homebrew doesn't automatically launchctl bootout when the service previously stopped unexpectedly (or never fully starts up, in this case)? So it tries to execute launchctl bootstrap for a service that's still "active", resulting in that particular error. While technically this makes perfect sense, it's a bit inconsistent with the output of brew services saying it's stopped. You would expect simply brew services start ... to be sufficient.

In case it matters: the version of my homebrew-services is commit 34637816b6d34a6f9414758ceaa410437c4e997c, which should be latest master.

MikeMcQuaid commented 2 weeks ago

brew services has never tried to cleanup PIDs on a crash. It would be a good feature request for it to do so, if that's what you're proposing, but to set expectations: it might be a while before anyone who isn't you picks this up and implements it.

Also, yes, Homebrew doesn't automatically run launchctl bootout for you.

brew services is a relatively thin wrapper (on macOS) around launchctl so it doesn't really "automatically" do very much at all.

GottemHams commented 2 weeks ago

it might be a while before anyone [...] picks this up

That's cool, at least I know why it won't start and can get it working again relatively easily.