Removing hiawatha removes /usr/bin/sv

[jmattsson]

Something appears broken in the package handling, as removing hiawatha also removes /usr/bin/sv! This is on the devuan-runit-2019-05-07 AMI:

root@ip-172-30-0-177:~# apt-get purge hiawatha
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  hiawatha*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 1,232 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 50049 files and directories currently installed.)
Removing hiawatha (10.9) ...
ok: down: hiawatha: 25s, normally up
Processing triggers for man-db (2.8.5-2) ...
(Reading database ... 50009 files and directories currently installed.)
Purging configuration files for hiawatha (10.9) ...
dpkg: warning: while removing hiawatha, directory '/var/log/hiawatha' not empty so not removed
dpkg: warning: while removing hiawatha, directory '/var/lib/hiawatha' not empty so not removed
root@ip-172-30-0-177:~# sv
bash: /usr/bin/sv: No such file or directory

Similarly, the same thing happens when removing some, but not all, other packages (e.g. incron).

I don't see /usr/bin/sv listed in any of the /var/lib/dpkg/info/*.list files, so I'm at a loss as to why it gets nuked. Does it have something to do with the packages runit, runit-init and runit-sysv being marked as removed?

Edit: Ah, this would be why:

# ls -l /etc/init.d/hiawatha
lrwxrwxrwx 1 root root 11 Feb 20 23:56 /etc/init.d/hiawatha -> /usr/bin/sv

[cloux]

Yes, that is why. It is a sysvinit compatibility feature of runit to have /etc/init.d/SCRIPT symlinked to /usr/bin/sv. Unfortunately, it seems that package management removes the target of the symlink instead of just the symlink itself. I am working on a solution, that temporarily removes the symlink while dpkg is in operation. It is a bit tricky to do. Temporarily, do not uninstall any services having symlinks in /etc/init.d.

If you want to remove hiawatha or any other service, just run svdeactivate hiawatha to disable it.

[jmattsson]

I ended up nuking all of /etc/init.d (and /etc/rc?.d) - I really don't need any sysv compatibility :)

But yeah, it's really surprising that it's the target that gets removed and not the symlink itself. Or that dpkg doesn't notice/care about the fact that the type has changed from file to symlink.

Good luck with the fix, and thanks for providing this AMI!

[cloux]

Don't nuke it just yet, that might cause all sorts of problems. The Debian/Devuan dpkg system including scripts like insserv or update-rc.d expect these directories and files to be present. Many prerm and postinst actions in the packages will fail without it, installations and upgrades will break. I would like to purge the init.d and rc*d directories myself, but I have to provide this compatibility layer to keep packages happy. To make sure the init.d symlinks are there, many of my runscripts even create them when the service starts, see here (acpid) or here (cron).

TIP: if you experiment with stuff like "what happens if I delete this directory", you might try my Vagrant box instead. It is easier to use and recycle broken instances than AWS, works locally, and obviously at no cost. You can experiment as long as you want, break stuff, and pay nothing.

TIP2: if runit gets damaged, like if sv gets accidentally removed, run sin runit-init. This will repair/update all runit binaries and runscripts.

And thanks for the feedback, it's good to know that other people find my AMI useful too :)

[jmattsson]

Solid advice for the typical environment, for sure. In my case it's fine because I don't actually want to have any other "regular" services fired up, so no pain expected due to lack of /etc/init.d :) My purpose with this particular incarnation is as a minimal, reliable nodejs platform, and I'm definitely partial to runsv for such scenarios.

I did see that the run scripts tended to set up the symlinks, and also noticed that they'd silently ignore doing so if there was not /etc/init.d around, so I took advantage of that!

TIP2 would've saved me a bit of time, but I very quickly learned that I wanted a backup copy of /usr/bin/sv stashed away on the side!

[cloux]

Nodejs (micro)services die easily on any unhandled exception, they almost beg to be supervised for reliability. This is a very good use case. A stripped thin OS that just boots into a supervisor as quickly as possible should do nicely.

With a twist: if you remove /etc/init.d, system upgrades will break. Running apt-get update && apt-get full-upgrade will produce an error when services are involved. I would not recommend to give up the upgrade capability. For security reasons, services like SSH should be updated. (I have a feeling that you understand the implications, just wanted to put a big fat WARNING sign here). For removing stuff, you can get some inspiration from ec2-instance-bootstrap. I also have a hard clean-wipe script to reduce the system to a bare minimum, let me know if you are interested, it's not published.

TIP2 info: the simple instaler (sin) already keeps a runit backup copy for you, the binaries are in /usr/src/runit/runit-2.1.2/command/.

TOPIC: I did some fixing and testing, and the sv binary should not get deleted anymore (AMI releases from version 5.1.3 and above). However, I am not entirely sure it's fixed, I will run some more tests. Please report if it happens again. To update, use latest AMI image or run sin runit-init in a running instance.

[cloux]

This issue is finally fixed by commits https://github.com/cloux/runit-base/commit/0e793e5c59d1938dbb78597a0cb20a003e508c75 , https://github.com/cloux/sin/commit/48afee7ba945be7a3af264ff183e2601dfa43d6b and https://github.com/cloux/sin/commit/6da35885ae404ecbdb2fc9df074cf16c6775f84f integrated in AWS from v5.1.9 and above.