saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

sls with empty output leads to "requisites were not found" #30971

Open skandyla opened 8 years ago

skandyla commented 8 years ago

Hi,

In this example I have two states, which retrieve some data from pillar. One state is depend from another. The error occurs when the first state doesn't get any data from pillar, so it produces zero output, and the second state fails with The following requisites were not found: require: sls:

          ID: custompkgs_httpd
    Function: pkg.installed
        Name: httpd
      Result: False
     Comment: The following requisites were not found:
                                 require:
                                     sls: repos.custom

The first state: salt/repos/custom.sls:

{% if grains['os'] == 'CentOS' %}                                                    
{% for yumrepo in salt['pillar.get']('yumrepos',{}) %}                               
{{ yumrepo }}_repo:                                                                  
  file.managed:                                                                      
    - name: /etc/yum.repos.d/{{ yumrepo }}.repo                                      
    - contents_pillar: yumrepos:{{ yumrepo }}                                        
    - mode: 600                                                                      
    - user: root                                                                     
{% endfor %}                                                                         
{% endif %}   

The second state: salt/packages/init.sls:

include:
  - repos.custom

{% for package, status in pillar.get('custompkgs', {}).items() %}                    
custompkgs_{{ package }}:                                                            
  pkg:                                                                               
    - name: {{ package }}                                                            
    - {{ status }}
    - require:
      - sls: repos.custom
{% endfor %} 

The second state is depend from the first, because I quite often deploy my private repositories and install custom packages which are depend from that private repositories.

Test's pillar data for the host:

custompkgs:                                                                          
  httpd: installed

When the first state (repos.custom) provide some output (I mean {% for package, status in pillar.get('custompkgs', {}).items() %} ) even without any changes, everything works fine.

Fox example, if I add some data to the pillar:

yumrepos:                                                                           
  tmp: |                                                                            
    #tmp 

everything works good.

Versions:

# salt-ssh --versions
Salt Version:
           Salt: 2015.8.5

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.3.1.el7.x86_64
         system: CentOS Linux 7.2.1511 Core
jfindlay commented 8 years ago

@skandyla, thanks for reporting. Since your first state expects pillar data by using contents_pillar, I would expect it to fail to render and thus not be available for the second state. Do you have an alternative expectation?

skandyla commented 8 years ago

@jfindlay, actually contents_pillar is another issue there, not related to the bug, I'm talking about. Sure the state rendering fails if contents_pillar is absent. But in this example this state does a job only if pillar data yumrepos: exist. So, execution of a module doesn't pass to the inside of a loop, where contents_pillar is defined.

The first state works with or without any content in pillar. It's some kind of state, which are applied to all of my hosts (70+). In case some data is defined at pillar, the state does a job. In other case, it does nothing. And it actually working that way.

salt/repos/custom.sls (short version)

{% for yumrepo in salt['pillar.get']('yumrepos',{}) %}
# do a job
{% endfor %}  

Example of launching this state for some host without any related data at pillar:

# time salt-ssh --wipe "example.com" state.sls repos.custom
example.com:

Summary for example.com
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1

# time salt-ssh --wipe  "example.com" pillar.get yumrepos
example.com:

The problem is the second state. salt/packages/init.sls:

include:
  - repos.custom

{% for package, status in pillar.get('custompkgs', {}).items() %}                    
custompkgs_{{ package }}:                                                            
  pkg:                                                                               
    - name: {{ package }}                                                            
    - {{ status }}
    - require:
      - sls: repos.custom
{% endfor %} 

This state shows an error:

...
     Comment: The following requisites were not found:
                                 require:
                                     sls: repos.custom

But the repos.custom state is working fine on himself, it just do nothing in case of absent pillar data. The problem is require: - sls: repos.custom doesn't understand that.

Finally, I've get it working buy introducing some function outside for loop:

{% if grains['os'] == 'CentOS' %}                                                    

# this is dummy function. yum-utils is already present at my system.
# But this forces the state to produce some response code
# in case of absent pillar data at 'yumrepos'
yum-utils:    
  pkg.installed                                                                                       

{% for yumrepo in salt['pillar.get']('yumrepos',{}) %}                               
{{ yumrepo }}_repo:                                                                  
  file.managed:                                                                      
    - name: /etc/yum.repos.d/{{ yumrepo }}.repo                                      
    - contents_pillar: yumrepos:{{ yumrepo }}                                        
    - mode: 600                                                                      
    - user: root                                                                     
{% endfor %}                                                                         
{% endif %} 

In that case, the second state (packages/init.sls) is working good. It's find the require dependence to be resolved. (There are also no data at pillar) But I don't think that this is a good solution.

I suppose there is some bug in sls rendering subsystem.

jfindlay commented 8 years ago

@skandyla, thanks for the extra explanation.

kiemlicz commented 6 years ago

Also suffering from this bug Do you have any updates about this?

OrangeDog commented 5 years ago

I think I've got a similar issue. This is including https://github.com/saltstack-formulas/letsencrypt-formula via gitfs.

include:
  - letsencrypt

...
    - require:
      - sls: letsencrypt
    Comment: The following requisites were not found:
                                require:
                                    sls: letsencrypt

Salt 2018.3.4

Using sls: 'letsencrypt.*' works though.

gigi206 commented 5 years ago

Same problem here on version 2019.2.0

If include an sls that contains only include I have also the message the following requisites were not found :

include:
  - .include1
  - .include2

I see this is a very old bug always present in the last version :'(

The workaround I use is to put inside the sls:

# https://github.com/saltstack/salt/issues/30971
bug:
  test.configurable_test_state:
    - name: workaround
    - changes: False
    - result: True
    - comment: Cf https://github.com/saltstack/salt/issues/30971
banholzer commented 2 years ago

can we expect any movement on this bug at any time? This is open quite long and nothing moved except assignments and labels.

mdschmitt commented 2 years ago

Just ran across this. "requisites not found" is a really non-obvious error to use for a simple "you required a file that said there's nothing to do" case. Tracking down and fixing took much longer than necessary based on this error. Even staring at the master's debug output, I was questioning my sanity because the file was for sure present, but still failing.

Happened with https://github.com/saltstack-formulas/prometheus-formula for me. If you happen to write a grain with prometheus: as the key and it has just one or two unrelated-to-the-formula values in it, config.get will happily use that + the defaults.

https://github.com/saltstack-formulas/prometheus-formula/blob/e03e7c9b20fb6760aa5162b55213a902b94e4888/prometheus/package/install.sls renders out to require https://github.com/saltstack-formulas/prometheus-formula/blob/e03e7c9b20fb6760aa5162b55213a902b94e4888/prometheus/package/repo/install.sls

...but if you're on RedHat and have only values from defaults.yaml + a couple no-op items in a grain set, the if condition on line 8 fails and therefore the file is "empty" and results in requisites were not found getting spit back from the prometheus/package/install.sls file

1 # -*- coding: utf-8 -*-
2 # vim: ft=sls
3
4 {%- if grains.os_family == 'RedHat' %}
5     {%- set tplroot = tpldir.split('/')[0] %}
6     {%- from tplroot ~ "/map.jinja" import prometheus as p with context %}
7
8     {%- if p.pkg.use_upstream_repo and 'repo' in p.pkg and p.pkg.repo %}
9         {%- from tplroot ~ "/files/macros.jinja" import format_kwargs with context %}
10
11 prometheus-package-repo-install-pkgrepo-managed:
12   pkgrepo.managed:
13     {{- format_kwargs(p.pkg.repo) }}
14
15     {%- endif %}
16 {%- else %}
17
18 prometheus-package-repo-install-pkgrepo-managed:
19   test.show_notification:
20     - name: Skipping repository configuration
21     - text: |
22         At the moment, there's no repo for {{ grains['os'] }}
23         See https://prometheus.io/download/
24
25 {%- endif %}

Took me forever to diagnose this considering it's a very simple case. The error is at the very least "misleading," if not actively incorrect.

Been a bug for 6+ years... @sagetherage , you're the maintainer that touched it last, do you know how we go about getting this worked out?