Closed SLoeuillet closed 3 years ago
Forget it : missing AF_PACKET in RestrictAddressFamilies
Could be cool to provide that kind of config file as other people could be interrested in running keepalived as non root/dyn user
I would be happy to include the above config file, but we would need to test that all of keepalived's functionality can work as a non-root user, and keepalived does rather a lot of things that normally require root privileges. I note for example that you do not have any IPVS configuration; would that require any other capabilities?
Does RestrictAddressFamilies also require AF_INET6?
I'll test your setup with a fairly complete configuration and see what I find, but I may need some help with changing the configuration if some things don't work, since I am no great expert in systemd configuration.
Well, instead of giving a full .services file that won't cover everything or that would be so large that it would mean root in disguise, perhaps a way to document that VRRP needs AF_RAW & AF_PACKET + CAP_NET_RAW and same for each sub-daemon/functionnality ?
and of course, if IPv6, AF_INET6, ...
Anyway my main problem here is that it segfaults with no way to catch the root cause
I have run your configuration on Ubuntu 21.04, with the systemd service file above, adding AF_PACKET as you identified, and also adding AF_INET6 because the BFD process uses AF_INET6 sockets regardless of whether it is actually using IPv4 or IPv6.
I am not not getting any error and keepalived is successfully running. Both VRRP and BFD are successfully communicating with another physical machine.
Oh, I suppose that's why I had the error 97 with BFD, because of missing AF_INET6 I did ignore it but good to know that BFD wants to bind to :: I suppose we can't disable Ipv6 at kernel boot so ? (ipv6.disable=1 in /etc/default/grub GRUB_CMDLINE_LINUX)
Even with AF_INET6 I still have those in the log :+1:
Oct 12 13:53:59 gobgpd00-bgp-ovh Keepalived[3383751]: Starting BFD child process, pid=3383758
Oct 12 13:53:59 gobgpd00-bgp-ovh Keepalived_bfd[3383758]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 12 13:53:59 gobgpd00-bgp-ovh Keepalived_bfd[3383758]: socket() error 97 (Address family not supported by protocol)
Oct 12 13:53:59 gobgpd00-bgp-ovh Keepalived_bfd[3383758]: scheduler: Error performing control on EPOLL instance (Bad file descriptor)
Oct 12 13:53:59 gobgpd00-bgp-ovh Keepalived_bfd[3383758]: scheduler: Cant register read event for fd [-1](Bad file descriptor)
Confirmed. Needed to re-enable IPv6 in kernel boot params to have BFD working well
Oct 12 13:58:21 gobgpd01-bgp-ovh Keepalived[905]: Starting BFD child process, pid=914
Oct 12 13:58:21 gobgpd01-bgp-ovh Keepalived_bfd[914]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 12 13:58:25 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Detection time is 5000 ms (was 0 ms)
Oct 12 13:58:26 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Detection time is 50 ms (was 5000 ms)
Oct 12 13:58:26 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Entering Up state
Oct 12 13:58:26 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Expired after 50 ms (24 usec overdue)
Oct 12 13:58:26 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Entering Down state (Local diagnostic - Control Detection Time Expired, Remote diagnostic - No Diagnostic)
Oct 12 13:58:27 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Detection time is 5000 ms (was 50 ms)
Oct 12 13:58:27 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Entering Up state
Oct 12 13:58:27 gobgpd01-bgp-ovh Keepalived_bfd[914]: BFD_Instance(BFD1) Detection time is 50 ms (was 5000 ms)
I have discovered that editing /etc/apport/crashdb.conf to comment out the line 'problem_types': ['Bug', 'Package'],
means that crash dump information is saved to /var/crash. Using that I have identified and resolved the segfault in the VRRP process that I expect is what you have been experiencing.
The BFD requirement for IPv6 was resolved in commits c1a41c3, 5a1f1e8 and d38103a a month or two ago, and so with the latest code the BFD process can now run without IPv6 if it is only using IPv4.
I have done some testing of all the features of keepalived that I can think of that might need various privileges/capabilities
To allow keepalived to load modules ip_vs (needed for IPVS configuration) and xt_set (needed for using iptables/ipset) Add: Ambient_Capabilities=CAP_SYS_MODULE CapabilityBoundingSet=CAP_SYS_MODULE ProtectKernelModules=no alternatively add a file in /usr/lib/modules-load.d with ip_vs and xt_set
To allow keepalived to adjust its realtime scheduling priority if it is not being scheduled fast enough RestrictRealtime=no
To allow keepalived to run with standard scheduling Add AmbientCapabilities=CAP_SYS_NICE CapabilityBoundingSet=CAP_SYS_NICE Remove CPUSchedulingPolicy=rr
The normal signal for stopping keepalived is SIGTERM KillSignal=SIGTERM
Allow keepalived to set sysctl values (needed for using VMACs) ProtectKernelTunables=no
Allow keepalived to set the UID/GID for scripts it runs AmbientCapabilities=CAP_KILL AmbientCapabilities=CAP_SETUID AmbientCapabilities=CAP_SETGID CapabilityBoundingSet=CAP_KILL CapabilityBoundingSet=CAP_SETUID CapabilityBoundingSet=CAP_SETGID
Allow keepalived to change owner/group of notify FIFO AmbientCapabilities=CAP_CHOWN CapabilityBoundingSet=CAP_CHOWN
Allow keepalived to call setrlimit for number of open files and coredump size AmbientCapabilities=CAP_SYS_RESOURCE CapabilityBoundingSet=CAP_SYS_RESOURCE
I haven't been able to get SNMP working with keepalived when running as a non-root user, so any help with that would be appreciated.
I think what we should do is add a keepalived-non-root.service file with appropriate comments relating to what each capability is needed for, so that people can customise the service file according to the needs of their particular configuration. I would like the default to include everything, to avoid issue reports being raised because people haven't enabled certain capabilities. @SLoeuillet what do you think?
It might be that we can add functionality to drop capabilities with keepalived for capabilities that are not required, based on the actual keepalived configuration in use.
There are some other issues relating to keepalived executing scripts (vrrp notify, notify_fifo_scripts, vrrp track scripts and CHECK_MISC checker scripts). keepalived is currently written assuming that it is running as root, and the code for handling scripts, especially if the user/group for running the scripts are specified, will not work. I am currently working on a patch for this.
Good to know it has been fixed upstream Didn't try master, only 2.0.19 from ubuntu focal distro Feel free to close or keep it open regarding Capabilities & co needed for each component (doc)
Yeah, providing a full non-root .services file could be good People would be able to remove unneeded CAPs at will instead of adding them as guess-work from root one
Commit 557f4e3 adds keepalived-non-root.service.in which produces keepalivee-non-root.service when make is run. Commit 461922b resolves a segfault that you were probably experiencing.
@SLoeuillet Many thanks for all your work doing the difficult part of this.
Describe the bug On ubuntu 20.04, using keepalived 2.0.19 from distribution but with custom .services file to run as a dynamic user, with CAP_NET_ADMIN, CAP_NET_BIND_SERVICE, CAP_NET_RAW
keepalived_vrrp segfaults in a loop, until end of time
To Reproduce Start the service, wait 4 seconds max. Crashes even if second node keepalived is not started
Expected behavior Having keepalived working as non-root user or at least telling me what it wants instead of crashing.
Keepalived version
Distro (please complete the following information):
Details of any containerisation or hosted service (e.g. AWS) Running on an OpenStack VM on OVH
Configuration file:
Notify and track scripts
System Log entries
Did keepalived coredump?
Additional context
Startup script