Clarification on how `mod_aggregate` works

[lorengordon]

I've written a custom state module with a mod_aggregate function and gotten it to work. It does indeed aggregate all the state data into one state and it executes that state with the aggregated data, as expected. However, it is also still executing all the states from which data was aggregated, which I did not expect. I thought mod_aggregate would aggregate all the state data and execute the function one time.

I fiddled with my mod_aggregate function for a while, debugging the inputs and attempting to influence the outputs, but I couldn't figure out how to get it to stop executing states that had been aggregated.

Can anyone please clarify how mod_aggregate is supposed to work, and whether it is possible to make it work that the way I expected?

Doc link: https://docs.saltstack.com/en/latest/ref/states/aggregate.html Cross post to user group: https://groups.google.com/d/msg/salt-users/iS82msSo4os/roE0Zzd0BwAJ

[Ch3LL]

@lorengordon I think showing the pkg example might point you in the right direction. I did some testing with a the following in the minion config:

➜  ~ grep -i aggreg /etc/salt/minion
state_aggregate: True

*Given an sls file like so:

➜  ~ cat /srv/salt/issues/33354.sls
lamp_stack:
  pkg.installed:
    - pkgs:
      - php
      - MySQL-python

memcached:
  pkg.installed:
    - name: memcached  

Then when i run the state file I'm only seeing the lamp_stack state being run:

[INFO    ] Executing state pkg.installed for lamp_stack
[INFO    ] Executing command ['rpm', '-qa', '--queryformat', '%{NAME}_|-%{EPOCH}_|-%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-(none)\n'] in directory '/root'
[INFO    ] Executing command ['repoquery', '--plugins', '--queryformat', '%{NAME}_|-%{ARCH}', '--pkgnarrow=all', '--all'] in directory '/root'
[INFO    ] Executing command ['yum', '--quiet', 'clean', 'expire-cache'] in directory '/root'
[DEBUG   ] output: 
[INFO    ] Executing command ['yum', '--quiet', 'check-update'] in directory '/root'
[INFO    ] Executing command ['yum', '-y', 'install', 'php', 'memcached', 'MySQL-python'] in directory '/root'
[INFO    ] Executing command ['rpm', '-qa', '--queryformat', '%{NAME}_|-%{EPOCH}_|-%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-(none)\n'] in directory '/root'
[INFO    ] Made the following changes:
'php' changed from 'absent' to '5.4.16-36.1.el7_2.1'
'memcached' changed from 'absent' to '1.4.15-9.el7'
'MySQL-python' changed from 'absent' to '1.2.3-11.el7'

If you look at this line is a great example of how to handle this to ensure the state is not run again if it already has. The mod_aggregate function is called each time but then bails out if it has already run.

Does that point you in the right direction?

[lorengordon]

@Ch3LL, I do have that line in my mod_aggregate function. Perhaps the salt.utils.gen_state_tag(chunk) function is somehow not accounting for custom state modules?

[lorengordon]

@Ch3LL, also, I was using aggregate: True in my state definition, not in the minion config file. Perhaps that has something to do with it?

[cachedout]

@lorengordon Oh, you need it in your minion config I think. I haven't checked but if memory serves, we don't forward scan for that before we begin processing.

[lorengordon]

@cachedout, the docs definitely say that you can use aggregate: True in the state definition, and it does not say there is a difference in the implementation between the two. Personally, I prefer using the state definition, as it feels easier to control than the minion config file, especially if I don't want every such state to aggregate. So, would you tend to consider this an issue in the docs, or a bug in the implementation?

[lorengordon]

I took another look at this, and even using the example state above, I still see the behavior I described in the opening post. It doesn't matter whether I use state_aggregate: True in the minion config, or use aggregate: True in the sls.

cat /srv/salt/states/base/test.sls:

lamp_stack:
  pkg.installed:
    - pkgs:
      - php
      - MySQL-python

memcached:
  pkg.installed:
    - name: memcached

grep -i aggreg /etc/salt/minion:

# Automatically aggregate all states that have support for mod_aggregate by
# aggregate just those types.
# state_aggregate:
#state_aggregate: False
state_aggregate: True

salt-call --local state.sls test:

[INFO    ] Running state [lamp_stack] at time 13:34:15.673200
[INFO    ] Executing state pkg.installed for lamp_stack
[INFO    ] Executing command ['rpm', '-qa', '--queryformat', '%{NAME}_|-%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-(none)\n'] in directory '/root'
[INFO    ] Executing command "repoquery --plugins --queryformat '%{NAME}_|-%{ARCH}'  --pkgnarrow=all --all" in directory '/root'
[INFO    ] Executing command 'yum -q clean expire-cache   ' in directory '/root'
[INFO    ] Executing command 'yum -q check-update   ' in directory '/root'
[INFO    ] Executing command 'yum -y     install php memcached MySQL-python' in directory '/root'
[INFO    ] Executing command ['rpm', '-qa', '--queryformat', '%{NAME}_|-%{VERSION}_|-%{RELEASE}_|-%{ARCH}_|-(none)\n'] in directory '/root'
[INFO    ] Made the following changes:
'perl-Module-Pluggable' changed from 'absent' to '3.90-141.el6_7.1'
'httpd-tools' changed from 'absent' to '2.2.15-53.el6.centos'
'apr' changed from 'absent' to '1.3.9-5.el6_2'
'perl-libs' changed from 'absent' to '5.10.1-141.el6_7.1'
'apr-util-ldap' changed from 'absent' to '1.3.9-3.el6_0.1'
'apr-util' changed from 'absent' to '1.3.9-3.el6_0.1'
'perl-Pod-Escapes' changed from 'absent' to '1.04-141.el6_7.1'
'perl-Pod-Simple' changed from 'absent' to '3.13-141.el6_7.1'
'perl-version' changed from 'absent' to '0.77-141.el6_7.1'
'MySQL-python' changed from 'absent' to '1.2.3-0.3.c1.1.el6'
'libevent' changed from 'absent' to '1.4.13-4.el6'
'php' changed from 'absent' to '5.3.3-47.el6'
'mailcap' changed from 'absent' to '2.1.31-2.el6'
'httpd' changed from 'absent' to '2.2.15-53.el6.centos'
'perl' changed from 'absent' to '5.10.1-141.el6_7.1'
'php-cli' changed from 'absent' to '5.3.3-47.el6'
'php-common' changed from 'absent' to '5.3.3-47.el6'
'memcached' changed from 'absent' to '1.4.4-3.el6'
[INFO    ] Loading fresh modules for state activity
[INFO    ] Completed state [lamp_stack] at time 13:34:40.626170
[INFO    ] Running state [memcached] at time 13:34:40.629661
[INFO    ] Executing state pkg.installed for memcached
[INFO    ] Package memcached is already installed
[INFO    ] Completed state [memcached] at time 13:34:40.630434
local:
----------
          ID: lamp_stack
    Function: pkg.installed
      Result: True
     Comment: 3 targeted packages were installed/updated.
     Started: 13:34:15.673200
    Duration: 24952.97 ms
     Changes:
              ----------
              MySQL-python:
                  ----------
                  new:
                      1.2.3-0.3.c1.1.el6
                  old:
              apr:
                  ----------
                  new:
                      1.3.9-5.el6_2
                  old:
              apr-util:
                  ----------
                  new:
                      1.3.9-3.el6_0.1
                  old:
              apr-util-ldap:
                  ----------
                  new:
                      1.3.9-3.el6_0.1
                  old:
              httpd:
                  ----------
                  new:
                      2.2.15-53.el6.centos
                  old:
              httpd-tools:
                  ----------
                  new:
                      2.2.15-53.el6.centos
                  old:
              libevent:
                  ----------
                  new:
                      1.4.13-4.el6
                  old:
              mailcap:
                  ----------
                  new:
                      2.1.31-2.el6
                  old:
              memcached:
                  ----------
                  new:
                      1.4.4-3.el6
                  old:
              perl:
                  ----------
                  new:
                      5.10.1-141.el6_7.1
                  old:
              perl-Module-Pluggable:
                  ----------
                  new:
                      3.90-141.el6_7.1
                  old:
              perl-Pod-Escapes:
                  ----------
                  new:
                      1.04-141.el6_7.1
                  old:
              perl-Pod-Simple:
                  ----------
                  new:
                      3.13-141.el6_7.1
                  old:
              perl-libs:
                  ----------
                  new:
                      5.10.1-141.el6_7.1
                  old:
              perl-version:
                  ----------
                  new:
                      0.77-141.el6_7.1
                  old:
              php:
                  ----------
                  new:
                      5.3.3-47.el6
                  old:
              php-cli:
                  ----------
                  new:
                      5.3.3-47.el6
                  old:
              php-common:
                  ----------
                  new:
                      5.3.3-47.el6
                  old:
----------
          ID: memcached
    Function: pkg.installed
      Result: True
     Comment: Package memcached is already installed
     Started: 13:34:40.629661
    Duration: 0.773 ms
     Changes:

Summary for local
------------
Succeeded: 2 (changed=1)
Failed:    0
------------
Total states run:     2

Versions report:

Salt Version:
           Salt: 2015.8.5

Dependency Versions:
         Jinja2: 2.2.1
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3c1
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 2.6.32-573.26.1.el6.x86_64
         system: CentOS 6.7 Final

[lorengordon]

I went ahead and updated salt to v2016.3.0 and am still seeing the same behavior.

[lorengordon]

@Ch3LL, @cachedout, any ideas? I did poke around the source code, and it seems the key is understanding salt/state.py and what is going on here and here, but I felt a little out of my depth with some of the salt internals.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.