search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.12k stars 5.47k forks source link

[BUG] job cache is saved locally even if job_cache is set to False in master config #66565

Open tomekrzymyszkiewicz opened 4 months ago

tomekrzymyszkiewicz commented 4 months ago

Description The salt master config file field job_cache: False does not apply in the work of the system. The jobs are being saved to the job cache at salt master even if job_cache: False is set up in salt master and salt syndic.

Setup There are 2 salt masters iabc-saltmaster-1 and oabc-saltmaster-1 , salt syndic abc-saltsyndic and minions (for example minion-123).

Salt masters config (iabc-saltmaster-1 and oabc-saltmaster-1 have same configs) The salt masters /etc/salt/master - part with returner and job cache:


job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass:  <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master

schedule:
  clean_old_jobs:
    run_on_start: true
    enabled: true
    function: pgjsonb.clean_old_jobs
    splay: 1800
    hours: 6

presence_events: True

The salt masters /etc/salt/minion

master:
 - <oabc-saltmaster-1 FQDN>
 - <iabc-saltmaster-1 FQDN>
metadata_server_grains: True

Salt syndic config (abc-saltsyndic) The salt syndic /etc/salt/master - part with returner and job cache:

job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass: <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master

The salt syndic /etc/salt/minion

master:
  - <iabc-saltmaster-1 FQDN>
  - <oabc-saltmaster-1 FQDN>

The machines are VMs running on cloud service with onedir packaging

Steps to Reproduce the behavior

  1. Find any minion assigned to syndic 
    root@abc-saltsyndic:~# salt-key -L
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] The functions from module 'key' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded key.output
Accepted Keys:
minion-123
Denied Keys:
Rejected Keys:
  2. Run job in the minion 
    root@abc-saltsyndic:~# salt 'minion-123' test.ping --async
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: abc-saltsyndic
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Closing AsyncReqChannel instance
Executed command with job ID: 20240521124445278489
  3. Check if job cache exists 
    
root@abc-saltsyndic:~# find /dev/shm/var/cache/salt/master/jobs -type f -mmin -1
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.minions.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.load.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid

root@abc-saltsyndic:~# cat /dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid 20240521124445278489

As a resuktl there is `jid` file with job ID of the job and in the directory are files `.minion.p` and `.load.p`.

**Expected behavior**
According to the docs ([Managing the Job Cache](https://docs.saltproject.io/en/latest/topics/jobs/job_cache.html)) there should be only `jid` file for checking for and preventing JID collisions. The `.minion.p` and `.load.p.
 should not be created.

**Versions Report**
<details><summary>on `iabc-saltmaster-1`</summary>

```yaml
Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 1.10.6
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04.5 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-169-generic
        system: Linux
       version: Ubuntu 20.04.5 focal

on `oabc-saltmaster-1` ```yaml Salt Version: Salt: 3006.5 Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: 1.10.6 gitdb: 4.0.11 gitpython: 3.1.40 Jinja2: 3.1.2 libgit2: 1.3.0 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: 2.6.1 pycryptodome: 3.9.8 pygit2: 1.7.0 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: 5.0.1 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: ubuntu 20.04.6 focal locale: utf-8 machine: x86_64 release: 5.4.0-150-generic system: Linux version: Ubuntu 20.04.6 focal ```
on `abc-saltsyndic` ```yaml Salt Version: Salt: 3006.5 Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.11 gitpython: 3.1.40 Jinja2: 3.1.2 libgit2: 1.3.0 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: 1.7.0 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: 5.0.1 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: ubuntu 20.04.5 focal locale: utf-8 machine: x86_64 release: 5.4.0-169-generic system: Linux version: Ubuntu 20.04.5 focal ```

Additional context Because of need to obfuscate sensitive data all values in triangular brackets <> have been described and the value is not shared.

welcome[bot] commented 4 months ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!