search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] CPU usage (FreeBSD kevent/kqueue) #61333

Open silenius opened 2 years ago

silenius commented 2 years ago

I have a FreeBSD host with many jails. In each jail a salt-minion is installed. Although each minion consumes something like 0.1% CPU it starts to have an impact on the load average.

Running truss -p <pid_of_minion> I noticed the following:

kevent(9,0x0,0,{ },1000,{ 0.640124878 })         = 0 (0x0)
getpid()                                         = 18245 (0x4745)
kevent(9,0x0,0,{ },1000,{ 0.958296299 })         = 0 (0x0)
getpid()                                         = 18245 (0x4745)
kevent(9,0x0,0,{ },1000,{ 0.963230610 })         = 0 (0x0)
getpid()                                         = 18245 (0x4745)
kevent(9,0x0,0,{ },1000,{ 0.973388911 })         = 0 (0x0)
getpid()                                         = 18245 (0x4745)
(...)

It happens every second (kevent followed by a getpid). I'm wondering if this could be the cause of CPU usage and why should it be run every second? Is it configurable?

This is on FreeBSD 13.0-RELEASE with:

chimay% salt --versions-report
Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.14.6
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.8.12 (default, Nov 27 2021, 21:04:30)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.3.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: freebsd 13.0 
        locale: utf-8
       machine: amd64
       release: 13.0-RELEASE-p4
        system: FreeBSD
       version: FreeBSD 13.0
garethgreenaway commented 2 years ago

@silenius Thanks for the report. What kind of CPU load are you seeing? Based on the above information this sounds like what you're seeing is the Salt interval loop which runs once a second. This is used by Salt components such as the scheduler and beacons. It is configurable with the loop_internval configuration option in the minion configuration.

silenius commented 2 years ago

I have a lot of jails and each salt-minion takes ~0.30% CPU. I tried to raise loop_interval from 1 to 10 but it doesn"t change anything (I've still a kevent(...) followed by a getpid() every second). (getpid() used to be slow under FreeBSD, I don't know if it's still the case)