search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.21k stars 5.48k forks source link

Pillar render fails but appears to have succeeded #50554

Closed pirogoeth closed 4 years ago

pirogoeth commented 6 years ago

Description of Issue/Question

My master consistently reports pillar render failures, but the trace log does not actually show any failure beyond the emission of the "data failed to compile" event. I have seen this with another pillar file that has no special rendering steps as well, so I will post that too.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

pillars/app/grainsrv/server.sls:

#!jinja|yaml|gpg
---

grainsrv:
  server:
    ssl:
      enabled: false
      certfile: /ssl/cert.pem
      keyfile: /ssl/key.pem
    container:
      image: myregistry.com/saltbox/grainsrv
      tag: v0.2.0
      mounts:
        data: /var/db/grainsrv
        ssl: "/etc/letsencrypt/live/{{ grains['fqdn'] }}"
      publish:
        http:
          host: 127.0.0.1
          port: 5000
      env:
        VIRTUAL_HOST: "{{ grains['fqdn'] }}"
      restart: always
    settings:
      auth:
        enabled: true
        username: REDACTED
        password: |
          -----BEGIN PGP MESSAGE-----

          REDACTED
          -----END PGP MESSAGE-----
      database_uri: sqlite:///data/grainsrv.sqlite
      debug: false
      swagger: true

pillars/app/consul/server.sls (other pillar with no special rendering beyond jinja):

---

consul:
  service: True
  user: consul
  group: consul
  version: 1.3.0
  config:
    # Base settings
    node_name: "{{ grains['fqdn'] }}"
    datacenter: "{{ grains['datacenter'] }}"
    data_dir: /var/lib/consul
    log_level: INFO
    ui: true

    # ACL settings
    acl_datacenter: "{{ grains['datacenter'] }}"
    acl_default_policy: deny
    acl_down_policy: extend-cache
    # TODO: Gen an agent token
    acl_agent_token: ""

    # Connect settings
    connect:
      enabled: false

    # Server settings
    server: True
    bind_addr: {{ '{{ GetInterfaceIP \"eth0\" }}' | yaml_squote }}
    bootstrap_expect: 3
    # TODO: Gen a gossip key
    encrypt: ""
    recursors:
      - 1.1.1.1
      - 1.0.0.1
      - 8.8.8.8

    # Telemetry settings
    telemetry:
      disable_hostname: true
      prometheus_retention_time: 96h

Even with safe_pillar_render_error set to False, there is no useful information logged either in the response or the master log.

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Output of pepper 'salt-master-n01.dc1.mydomain.com' state.highstate:

salt-master-n01.dc1.mydomain.com:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Rendering SLS 'app.grainsrv.server' failed. Please see master log for details.

Output of salt-master -l trace from start of highstate:

[TRACE   ] Clear payload received with command mk_token
[DEBUG   ] LazyLoaded pam.auth
[TRACE   ] Trying generic group list for 'sean'
[TRACE   ] Group list for user 'sean': ['salt', 'sean', 'ssh', 'sudo']
[DEBUG   ] LazyLoaded localfs.mk_token
[TRACE   ] IPCServer: Handling connection to address: 
[TRACE   ] Clear payload received with command publish
[TRACE   ] Compiled auth_list: [u'@wheel', u'@runner', u'@jobs', u'.*', u'@wheel', u'@runner', u'@jobs', u'*']
[DEBUG   ] Minion tokenized user = "sean"
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pull.ipc
[TRACE   ] IPCServer: Handling connection to address: 
[DEBUG   ] Sending event: tag = 20181117210925244993; data = {u'_stamp': '2018-11-17T20:09:25.246400', u'minions': [u'salt-master-n01.dc1.mydomain.com']}
[TRACE   ] get_event() received = {u'tag': '20181117210925244993', u'data': {u'_stamp': u'2018-11-17T20:09:25.246400', u'minions': [u'salt-master-n01.dc1.mydomain.com']}}
[TRACE   ] get_event() received = {u'tag': '20181117210925244993', u'data': {u'_stamp': u'2018-11-17T20:09:25.246400', u'minions': [u'salt-master-n01.dc1.mydomain.com']}}
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Gathering reactors for tag 20181117210925244993
[DEBUG   ] Sending event: tag = salt/job/20181117210925244993/new; data = {u'tgt_type': 'glob', u'jid': u'20181117210925244993', u'user': 'sean', u'tgt': 'salt-master-n01.dc1.mydomain.com', u'arg': [], u'fun': 'state.highstate', u'missing': [], u'_stamp': '2018-11-17T20:09:25.252424', u'minions': [u'salt-master-n01.dc1.mydomain.com']}
[TRACE   ] get_event() received = {u'tag': 'salt/job/20181117210925244993/new', u'data': {u'tgt_type': u'glob', u'jid': u'20181117210925244993', u'tgt': u'salt-master-n01.dc1.mydomain.com', u'missing': [], u'_stamp': u'2018-11-17T20:09:25.252424', u'user': u'sean', u'arg': [], u'fun': u'state.highstate', u'minions': [u'salt-master-n01.dc1.mydomain.com']}}
[DEBUG   ] Gathering reactors for tag salt/job/20181117210925244993/new
[DEBUG   ] Adding minions for job 20181117210925244993: [u'salt-master-n01.dc1.mydomain.com']
[INFO    ] User sean Published command state.highstate with jid 20181117210925244993
[DEBUG   ] Published command details {u'tgt_type': 'glob', u'jid': u'20181117210925244993', u'tgt': 'salt-master-n01.dc1.mydomain.com', u'ret': '', u'user': 'sean', u'arg': [], u'fun': 'state.highstate'}
[TRACE   ] get_event() received = {u'tag': 'salt/job/20181117210925244993/new', u'data': {u'tgt_type': u'glob', u'jid': u'20181117210925244993', u'tgt': u'salt-master-n01.dc1.mydomain.com', u'missing': [], u'_stamp': u'2018-11-17T20:09:25.252424', u'user': u'sean', u'arg': [], u'fun': u'state.highstate', u'minions': [u'salt-master-n01.dc1.mydomain.com']}}
[DEBUG   ] Signing data packet
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt.sign_message: Signing message.
[TRACE   ] Accepted unpacked package from puller
[TRACE   ] Sending ZMQ-unfiltered data over publisher tcp://[::]:4505
[TRACE   ] Unfiltered data has been sent
[TRACE   ] Getting data from puller ipc:///var/run/salt/master/publish_pull.ipc
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[TRACE   ] AES payload received with command _master_opts
[DEBUG   ] Re-using gitfs object for process 2647
[TRACE   ] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE   ] Master function call _master_opts took 0.00058388710022 seconds
[TRACE   ] AES payload received with command _file_envs
[TRACE   ] Master function call _file_envs took 5.69820404053e-05 seconds
[TRACE   ] AES payload received with command _file_list
[DEBUG   ] Re-using gitfs object for process 2646
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/gitfs/.base.w created
[TRACE   ] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[DEBUG   ] Including configuration from '/etc/salt/master.d/engines.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/engines.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/gitfs.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gitfs.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/logging.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt-master-n01.dc1.mydomain.com
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/gitfs/.base.w removed
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/roots/.base.w created
[TRACE   ] roots: Processing /srv/salt/engines
[TRACE   ] roots: /srv/salt/engines is not a link
[TRACE   ] roots: /srv/salt/engines relative path is engines
[TRACE   ] roots: Processing /srv/salt/_engines
[TRACE   ] roots: /srv/salt/_engines is not a link
[TRACE   ] roots: /srv/salt/_engines relative path is _engines
[TRACE   ] roots: Processing /srv/salt/reactor
[TRACE   ] roots: /srv/salt/reactor is not a link
[TRACE   ] roots: /srv/salt/reactor relative path is reactor
[TRACE   ] roots: Processing /srv/salt/_grains
[TRACE   ] roots: /srv/salt/_grains is not a link
[TRACE   ] roots: /srv/salt/_grains relative path is _grains
[TRACE   ] roots: Processing /srv/salt/orchestrate
[TRACE   ] roots: /srv/salt/orchestrate is not a link
[TRACE   ] roots: /srv/salt/orchestrate relative path is orchestrate
[TRACE   ] roots: Processing /srv/salt/engines/logdna.py
[TRACE   ] roots: /srv/salt/engines/logdna.py is not a link
[TRACE   ] roots: /srv/salt/engines/logdna.py relative path is engines/logdna.py
[TRACE   ] roots: Processing /srv/salt/_engines/logdna.py
[TRACE   ] roots: /srv/salt/_engines/logdna.py is not a link
[TRACE   ] roots: /srv/salt/_engines/logdna.py relative path is _engines/logdna.py
[TRACE   ] roots: Processing /srv/salt/_engines/logdna.pyc
[TRACE   ] roots: /srv/salt/_engines/logdna.pyc is not a link
[TRACE   ] roots: /srv/salt/_engines/logdna.pyc relative path is _engines/logdna.pyc
[DEBUG   ] File matching file_ignore_glob. Skipping: _engines/logdna.pyc
[TRACE   ] roots: Processing /srv/salt/reactor/notify_telegram.sls
[TRACE   ] roots: /srv/salt/reactor/notify_telegram.sls is not a link
[TRACE   ] roots: /srv/salt/reactor/notify_telegram.sls relative path is reactor/notify_telegram.sls
[TRACE   ] roots: Processing /srv/salt/reactor/sync_grains.sls
[TRACE   ] roots: /srv/salt/reactor/sync_grains.sls is not a link
[TRACE   ] roots: /srv/salt/reactor/sync_grains.sls relative path is reactor/sync_grains.sls
[TRACE   ] roots: Processing /srv/salt/reactor/update_fileserver.sls
[TRACE   ] roots: /srv/salt/reactor/update_fileserver.sls is not a link
[TRACE   ] roots: /srv/salt/reactor/update_fileserver.sls relative path is reactor/update_fileserver.sls
[TRACE   ] roots: Processing /srv/salt/orchestrate/update_fileserver.sls
[TRACE   ] roots: /srv/salt/orchestrate/update_fileserver.sls is not a link
[TRACE   ] roots: /srv/salt/orchestrate/update_fileserver.sls relative path is orchestrate/update_fileserver.sls
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/roots/.base.w removed
[TRACE   ] Master function call _file_list took 0.0994691848755 seconds
[TRACE   ] AES payload received with command _file_list
[DEBUG   ] Re-using gitfs object for process 2642
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/gitfs/.lookup-fix-3.w created
[TRACE   ] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/gitfs/.lookup-fix-3.w removed
[TRACE   ] Master function call _file_list took 0.00990796089172 seconds
[TRACE   ] AES payload received with command _file_list
[DEBUG   ] Re-using gitfs object for process 2638
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/gitfs/.revert-23-fix_service_name_for_sle11.w created
[TRACE   ] Returning env cache data from /var/cache/salt/master/gitfs/envs.p
[TRACE   ] Lockfile /var/cache/salt/master/file_lists/gitfs/.revert-23-fix_service_name_for_sle11.w removed
[TRACE   ] Master function call _file_list took 0.00741004943848 seconds
[TRACE   ] AES payload received with command _pillar
[DEBUG   ] Determining pillar cache
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/engines.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/engines.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Including configuration from '/etc/salt/master.d/gitfs.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gitfs.conf
[DEBUG   ] pygit2 git_pillar_provider enabled
[TRACE   ] Loaded git_pillar as virtual git
[DEBUG   ] LazyLoaded git.ext_pillar
[DEBUG   ] Including configuration from '/etc/salt/master.d/logging.conf'
[DEBUG   ] Current fetch URL for git_pillar remote 'master https://myvcs.com/saltbox/pillars.git': https://myvcs.com/saltbox/pillars.git (desired: https://myvcs.com/saltbox/pillars.git)
[DEBUG   ] Current refspecs for git_pillar remote 'master https://myvcs.com/saltbox/pillars.git': [u'+refs/heads/*:refs/remotes/origin/*', u'+refs/tags/*:refs/tags/*'] (desired: [u'+refs/heads/*:refs/remotes/origin/*', u'+refs/tags/*:refs/tags/*'])
[DEBUG   ] Current http.sslVerify for git_pillar remote 'master https://myvcs.com/saltbox/pillars.git': false (desired: false)
[DEBUG   ] git_pillar is processing pillar SLS from /var/cache/salt/master/git_pillar/base for pillar env 'base'
[DEBUG   ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[TRACE   ] Error loading grains.from_grainsrv: Could not determine grainsrv host from config
[TRACE   ] Error loading grains.napalm: "napalm"" (/usr/lib/python2.7/dist-packages/salt/grains/napalm.pyc) cannot be loaded: NAPALM is not installed or not running in a (proxy) minion
[TRACE   ] Loading core.append_domain grain
[TRACE   ] Loading core.default_gateway grain
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[TRACE   ] Loading core.dns grain
[TRACE   ] Loading core.get_machine_id grain
[TRACE   ] Loading core.get_master grain
[TRACE   ] Loading core.get_server_id grain
[TRACE   ] Loading core.hostname grain
[TRACE   ] Loading core.hwaddr_interfaces grain
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/top.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/master/git_pillar/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/top.sls' using 'jinja' renderer: 0.0108768939972
[DEBUG   ] Rendered data from file: /var/cache/salt/master/git_pillar/base/top.sls:
---

base:
  '*':
    - app.grainsrv.client

  'roles:salt_master':
    - match: grain
    - engines
    - hooks
    - saltbox
    - notifiers.telegram

  'roles:consul_agent':
    - match: grain
    - app.consul.agent

  'roles:consul_server':
    - match: grain
    - app.consul.server

  'roles:letsencrypt':
    - match: grain
    - app.letsencrypt

  'roles:docker_host':
    - match: grain
    - app.logspout

  'G@roles:docker_host and G@docker:tls:true':
    - match: compound
    - app.docker

  'G@roles:docker_host and G@roles:grainsrv':
    - match: compound
    - app.grainsrv.server
    - app.nginx_proxy

  'G@roles:docker_host and G@roles:saltbot':
    - match: compound
    - app.saltbot

  'salt-master-n01.dc1.mydomain.com':
    - targeted.salt_master_n01_dc1_mydomain.com

[TRACE   ] Loading core.id_ grain
[TRACE   ] Loading core.ip4_interfaces grain
[TRACE   ] Loading core.ip6_interfaces grain
[TRACE   ] Loading core.ip_fqdn grain
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'base', OrderedDict([(u'*', [u'app.grainsrv.client']), (u'roles:salt_master', [OrderedDict([(u'match', u'grain')]), u'engines', u'hooks', u'saltbox', u'notifiers.telegram']), (u'roles:consul_agent', [OrderedDict([(u'match', u'grain')]), u'app.consul.agent']), (u'roles:consul_server', [OrderedDict([(u'match', u'grain')]), u'app.consul.server']), (u'roles:letsencrypt', [OrderedDict([(u'match', u'grain')]), u'app.letsencrypt']), (u'roles:docker_host', [OrderedDict([(u'match', u'grain')]), u'app.logspout']), (u'G@roles:docker_host and G@docker:tls:true', [OrderedDict([(u'match', u'compound')]), u'app.docker']), (u'G@roles:docker_host and G@roles:grainsrv', [OrderedDict([(u'match', u'compound')]), u'app.grainsrv.server', u'app.nginx_proxy']), (u'G@roles:docker_host and G@roles:saltbot', [OrderedDict([(u'match', u'compound')]), u'app.saltbot']), (u'salt-master-n01.dc1.mydomain.com', [u'targeted.salt_master_n01_dc1_mydomain.com'])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/top.sls' using 'yaml' renderer: 0.0123028755188
[DEBUG   ] compound_match: salt-master-n01.dc1.mydomain.com ? *
[DEBUG   ] compound_match salt-master-n01.dc1.mydomain.com ? "*" => "True"
[DEBUG   ] grains target: roles:salt_master
[DEBUG   ] Attempting to match 'salt_master' in 'roles' using delimiter ':'
[DEBUG   ] grains target: roles:consul_agent
[DEBUG   ] Attempting to match 'consul_agent' in 'roles' using delimiter ':'
[DEBUG   ] grains target: roles:consul_server
[DEBUG   ] Attempting to match 'consul_server' in 'roles' using delimiter ':'
[DEBUG   ] grains target: roles:letsencrypt
[DEBUG   ] Attempting to match 'letsencrypt' in 'roles' using delimiter ':'
[DEBUG   ] grains target: roles:docker_host
[DEBUG   ] Attempting to match 'docker_host' in 'roles' using delimiter ':'
[DEBUG   ] compound_match: salt-master-n01.dc1.mydomain.com ? G@roles:docker_host and G@docker:tls:true
[DEBUG   ] grains target: roles:docker_host
[DEBUG   ] Attempting to match 'docker_host' in 'roles' using delimiter ':'
[DEBUG   ] grains target: docker:tls:true
[DEBUG   ] Attempting to match 'tls:true' in 'docker' using delimiter ':'
[DEBUG   ] Attempting to match 'true' in 'tls' using delimiter ':'
[DEBUG   ] Attempting to match 'true' in 'docker:tls' using delimiter ':'
[DEBUG   ] compound_match salt-master-n01.dc1.mydomain.com ? "G@roles:docker_host and G@docker:tls:true" => "True and False"
[DEBUG   ] compound_match: salt-master-n01.dc1.mydomain.com ? G@roles:docker_host and G@roles:grainsrv
[DEBUG   ] grains target: roles:docker_host
[DEBUG   ] Attempting to match 'docker_host' in 'roles' using delimiter ':'
[DEBUG   ] grains target: roles:grainsrv
[DEBUG   ] Attempting to match 'grainsrv' in 'roles' using delimiter ':'
[DEBUG   ] compound_match salt-master-n01.dc1.mydomain.com ? "G@roles:docker_host and G@roles:grainsrv" => "True and True"
[DEBUG   ] compound_match: salt-master-n01.dc1.mydomain.com ? G@roles:docker_host and G@roles:saltbot
[DEBUG   ] grains target: roles:docker_host
[DEBUG   ] Attempting to match 'docker_host' in 'roles' using delimiter ':'
[DEBUG   ] grains target: roles:saltbot
[DEBUG   ] Attempting to match 'saltbot' in 'roles' using delimiter ':'
[DEBUG   ] compound_match salt-master-n01.dc1.mydomain.com ? "G@roles:docker_host and G@roles:saltbot" => "True and False"
[DEBUG   ] compound_match: salt-master-n01.dc1.mydomain.com ? salt-master-n01.dc1.mydomain.com
[DEBUG   ] compound_match salt-master-n01.dc1.mydomain.com ? "salt-master-n01.dc1.mydomain.com" => "True"
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/app/grainsrv/client.sls
[DEBUG   ] LazyLoaded gpg.render
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'grainsrv', OrderedDict([(u'client', OrderedDict([(u'basic_auth', OrderedDict([(u'enabled', True), (u'username', u'REDACTED'), (u'password', u'-----BEGIN PGP MESSAGE-----\n\nREDACTED\n-----END PGP MESSAGE-----\n')])), (u'enabled', True), (u'port', 443), (u'use_fqdn', True), (u'use_https', True)]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/grainsrv/client.sls' using 'yaml' renderer: 0.00304007530212
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] Reading GPG keys from: /etc/salt/gpgkeys
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/grainsrv/client.sls' using 'gpg' renderer: 0.0329451560974
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/engines.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/master/git_pillar/base']
[TRACE   ] Loading core.ip_interfaces grain
[TRACE   ] Loading core.linux_distribution grain
[TRACE   ] Loading core.locale_info grain
[TRACE   ] Loading core.os_data grain
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/engines.sls' using 'jinja' renderer: 0.0142269134521
[DEBUG   ] Rendered data from file: /var/cache/salt/master/git_pillar/base/engines.sls:
---

engines:
  logdna:
    path: /srv/salt/_engines/logdna.py
    source: https://raw.githubusercontent.com/pirogoeth/logdna-saltstack/master/logdna.py
    checksum: sha256=77c3d074e6cebba9172d4e6ba9532a0728b0bb58b787b537e22a7d1c9cff958e

[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'engines', OrderedDict([(u'logdna', OrderedDict([(u'path', u'/srv/salt/_engines/logdna.py'), (u'source', u'https://raw.githubusercontent.com/pirogoeth/logdna-saltstack/master/logdna.py'), (u'checksum', u'sha256=77c3d074e6cebba9172d4e6ba9532a0728b0bb58b787b537e22a7d1c9cff958e')]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/engines.sls' using 'yaml' renderer: 0.00308203697205
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/hooks.sls
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'hooks', OrderedDict([(u'fileserver', OrderedDict([(u'secret', u'-----BEGIN PGP MESSAGE-----\n\nREDACTED\n-----END PGP MESSAGE-----\n')]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/hooks.sls' using 'yaml' renderer: 0.00312685966492
[DEBUG   ] Reading GPG keys from: /etc/salt/gpgkeys
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/hooks.sls' using 'gpg' renderer: 0.0339560508728
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/saltbox.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/master/git_pillar/base']
[TRACE   ] 'virt-what' could not be found in the following search path: '[u'/usr/local/sbin', u'/usr/local/bin', u'/usr/sbin', u'/usr/bin', u'/sbin', u'/bin']'
[DEBUG   ] Please install 'virt-what' to improve results of the 'virtual' grain.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/saltbox.sls' using 'jinja' renderer: 0.0143699645996
[DEBUG   ] Rendered data from file: /var/cache/salt/master/git_pillar/base/saltbox.sls:
---

saltbox:
  directories:
    - /srv/salt/orchestrate
    - /srv/salt/_engines

  orchestrator:
    update_fileserver:
      path: /srv/salt/orchestrate/update_fileserver.sls

  reactor:
    notify_telegram:
      path: /srv/salt/reactor/notify_telegram.sls

    sync_grains:
      path: /srv/salt/reactor/sync_grains.sls

    update_fileserver:
      path: /srv/salt/reactor/update_fileserver.sls

  master:
    api:
      port: 8000
      ssl_chain: /etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/chain.pem
      ssl_crt: /etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/cert.pem
      ssl_key: /etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/privkey.pem

    engines:
      engines:
        - logdna:
            ingestion_key: REDACTED
      engine_dirs:
        - /srv/salt/_engines
      module_dirs:
        - /var/cache/salt/master/extmods

    external_auth:
      modules:
        pam:
          sean:
            - '@wheel'
            - '@runner'
            - '@jobs'
            - '*'
          salt%:
            - '@wheel'
            - '@runner'
            - '@jobs'
            - '.*'

    fileserver:
      backend:
        - gitfs
        - roots
      default_top: base
      file_roots:
        base:
          - /srv/salt/
      top_file_merging_strategy: merge

    gitfs:
      passphrase: ~
      privkey: /etc/salt/ssh/gitfs
      provider: pygit2
      pubkey: /etc/salt/ssh/gitfs.pub
      refspecs:
        - '+refs/heads/*:refs/remotes/origin/*'
        - '+refs/tags/*:refs/tags/*'
      remotes:
        - https://myvcs.com/saltbox/states.git
        - https://github.com/jeduardo/nomad-formula.git
        - https://github.com/saltstack-formulas/consul-formula.git
        - https://github.com/saltstack-formulas/docker-formula.git
        - https://github.com/saltstack-formulas/letsencrypt-formula.git
        - https://github.com/saltstack-formulas/ntp-formula.git
        - https://github.com/saltstack-formulas/openssh-formula.git
        - https://github.com/saltstack-formulas/users-formula.git
        - https://github.com/saltstack-formulas/vault-formula.git

    pillar:
      ext_pillar_first: true
      git_pillar_base: master
      git_pillar_branch: master
      git_pillar_privkey: /etc/salt/ssh/gitfs
      git_pillar_provider: pygit2
      git_pillar_pubkey: /etc/salt/ssh/gitfs.pub
      git_pillar_refspecs:
        - '+refs/heads/*:refs/remotes/origin/*'
        - '+refs/tags/*:refs/tags/*'
      ext:
        - git:
          - master https://myvcs.com/saltbox/pillars.git:
            - name: base
            - user: REDACTED
            - password: REDACTED
      roots:
        base:
          - /srv/pillar

    reactor:
      - 'salt/minion/*/start':
        - /srv/salt/reactor/sync_grains.sls
      - 'salt/netapi/hook/fileserver/update':
        - /srv/salt/reactor/update_fileserver.sls
      - 'notify/telegram':
        - /srv/salt/reactor/notify_telegram.sls

[TRACE   ] Loading core.path grain
[TRACE   ] Loading core.pythonexecutable grain
[TRACE   ] Loading core.pythonpath grain
[TRACE   ] Loading core.pythonversion grain
[TRACE   ] Loading core.saltpath grain
[TRACE   ] Loading core.saltversion grain
[TRACE   ] Loading core.saltversioninfo grain
[TRACE   ] Loading core.zmqversion grain
[TRACE   ] Loading from_fqdn.main grain
[TRACE   ] Loading disks.disks grain
[TRACE   ] Device sr0 reports itself as an HDD
[TRACE   ] Device sda reports itself as an HDD
[TRACE   ] Loading extra.config grain
[DEBUG   ] Loading static grains from /etc/salt/grains
[TRACE   ] Loading extra.shell grain
[TRACE   ] Loading mdadm.mdadm grain
[TRACE   ] Loading minion_process.grains grain
[TRACE   ] Loading opts.opts grain
[TRACE   ] Loading zfs.zfs grain
[TRACE   ] 'zfs-fuse' could not be found in the following search path: '[u'/usr/local/sbin', u'/usr/local/bin', u'/usr/sbin', u'/usr/bin', u'/sbin', u'/bin']'
[TRACE   ] 'zpool' could not be found in the following search path: '[u'/usr/local/sbin', u'/usr/local/bin', u'/usr/sbin', u'/usr/bin', u'/sbin', u'/bin']'
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'saltbox', OrderedDict([(u'directories', [u'/srv/salt/orchestrate', u'/srv/salt/_engines']), (u'orchestrator', OrderedDict([(u'update_fileserver', OrderedDict([(u'path', u'/srv/salt/orchestrate/update_fileserver.sls')]))])), (u'reactor', OrderedDict([(u'notify_telegram', OrderedDict([(u'path', u'/srv/salt/reactor/notify_telegram.sls')])), (u'sync_grains', OrderedDict([(u'path', u'/srv/salt/reactor/sync_grains.sls')])), (u'update_fileserver', OrderedDict([(u'path', u'/srv/salt/reactor/update_fileserver.sls')]))])), (u'master', OrderedDict([(u'api', OrderedDict([(u'port', 8000), (u'ssl_chain', u'/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/chain.pem'), (u'ssl_crt', u'/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/cert.pem'), (u'ssl_key', u'/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/privkey.pem')])), (u'engines', OrderedDict([(u'engines', [OrderedDict([(u'logdna', OrderedDict([(u'ingestion_key', u'REDACTED')]))])]), (u'engine_dirs', [u'/srv/salt/_engines']), (u'module_dirs', [u'/var/cache/salt/master/extmods'])])), (u'external_auth', OrderedDict([(u'modules', OrderedDict([(u'pam', OrderedDict([(u'sean', [u'@wheel', u'@runner', u'@jobs', u'*']), (u'salt%', [u'@wheel', u'@runner', u'@jobs', u'.*'])]))]))])), (u'fileserver', OrderedDict([(u'backend', [u'gitfs', u'roots']), (u'default_top', u'base'), (u'file_roots', OrderedDict([(u'base', [u'/srv/salt/'])])), (u'top_file_merging_strategy', u'merge')])), (u'gitfs', OrderedDict([(u'passphrase', None), (u'privkey', u'/etc/salt/ssh/gitfs'), (u'provider', u'pygit2'), (u'pubkey', u'/etc/salt/ssh/gitfs.pub'), (u'refspecs', [u'+refs/heads/*:refs/remotes/origin/*', u'+refs/tags/*:refs/tags/*']), (u'remotes', [u'https://myvcs.com/saltbox/states.git', u'https://github.com/jeduardo/nomad-formula.git', u'https://github.com/saltstack-formulas/consul-formula.git', u'https://github.com/saltstack-formulas/docker-formula.git', u'https://github.com/saltstack-formulas/letsencrypt-formula.git', u'https://github.com/saltstack-formulas/ntp-formula.git', u'https://github.com/saltstack-formulas/openssh-formula.git', u'https://github.com/saltstack-formulas/users-formula.git', u'https://github.com/saltstack-formulas/vault-formula.git'])])), (u'pillar', OrderedDict([(u'ext_pillar_first', True), (u'git_pillar_base', u'master'), (u'git_pillar_branch', u'master'), (u'git_pillar_privkey', u'/etc/salt/ssh/gitfs'), (u'git_pillar_provider', u'pygit2'), (u'git_pillar_pubkey', u'/etc/salt/ssh/gitfs.pub'), (u'git_pillar_refspecs', [u'+refs/heads/*:refs/remotes/origin/*', u'+refs/tags/*:refs/tags/*']), (u'ext', [OrderedDict([(u'git', [OrderedDict([(u'master https://myvcs.com/saltbox/pillars.git', [OrderedDict([(u'name', u'base')]), OrderedDict([(u'user', u'REDACTED')]), OrderedDict([(u'password', u'REDACTED')])])])])])]), (u'roots', OrderedDict([(u'base', [u'/srv/pillar'])]))])), (u'reactor', [OrderedDict([(u'salt/minion/*/start', [u'/srv/salt/reactor/sync_grains.sls'])]), OrderedDict([(u'salt/netapi/hook/fileserver/update', [u'/srv/salt/reactor/update_fileserver.sls'])]), OrderedDict([(u'notify/telegram', [u'/srv/salt/reactor/notify_telegram.sls'])])])]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/saltbox.sls' using 'yaml' renderer: 0.0395638942719
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/notifiers/telegram.sls
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'telegram', OrderedDict([(u'chat_id', u'REDACTED'), (u'token', u'-----BEGIN PGP MESSAGE-----\n\nREDACTED\n-----END PGP MESSAGE-----\n')]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/notifiers/telegram.sls' using 'yaml' renderer: 0.00402617454529
[DEBUG   ] Reading GPG keys from: /etc/salt/gpgkeys
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/notifiers/telegram.sls' using 'gpg' renderer: 0.0317718982697
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/app/logspout.sls
[DEBUG   ] LazyLoaded local_cache.clean_old_jobs
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'logspout', OrderedDict([(u'container', OrderedDict([(u'image', u'gliderlabs/logspout'), (u'tag', u'latest'), (u'name', u'logspout'), (u'args', u'syslog+tls://REDACTED'), (u'env', OrderedDict([(u'ALLOW_TTY', u'true'), (u'DEBUG', u'false'), (u'HTTP_PORT', 8000), (u'LOGSPOUT_TLS_CA_CERTS', u'/etc/ld-root-ca.crt'), (u'LOGSPOUT_TLS_HARDENING', u'true'), (u'SYSLOG_STRUCTURED_DATA', u'-----BEGIN PGP MESSAGE-----\n\nREDACTED\n-----END PGP MESSAGE-----\n')])), (u'restart', u'always'), (u'publish', [u'127.0.0.1:28000:8000']), (u'mounts', [u'/etc/ld-root-ca.crt:/etc/ld-root-ca.crt:ro'])])), (u'settings', OrderedDict([(u'use_system_hostname', True)]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/logspout.sls' using 'yaml' renderer: 0.00952005386353
[DEBUG   ] Reading GPG keys from: /etc/salt/gpgkeys
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/logspout.sls' using 'gpg' renderer: 0.0340819358826
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/app/grainsrv/server.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/master/git_pillar/base']
[DEBUG   ] LazyLoaded localfs.list_tokens
[DEBUG   ] Set update lock for git_pillar remote 'master https://myvcs.com/saltbox/pillars.git'
[DEBUG   ] Fetching git_pillar remote 'master https://myvcs.com/saltbox/pillars.git'
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/grainsrv/server.sls' using 'jinja' renderer: 0.0104510784149
[DEBUG   ] Rendered data from file: /var/cache/salt/master/git_pillar/base/app/grainsrv/server.sls:
#!jinja|yaml|gpg
---

grainsrv:
  server:
    ssl:
      enabled: false
      certfile: /ssl/cert.pem
      keyfile: /ssl/key.pem
    container:
      image: myregistry.com/saltbox/grainsrv
      tag: v0.2.0
      mounts:
        data: /var/db/grainsrv
        ssl: "/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com"
      publish:
        http:
          host: 127.0.0.1
          port: 5000
      env:
        VIRTUAL_HOST: "salt-master-n01.dc1.mydomain.com"
      restart: always
    settings:
      auth:
        enabled: true
        username: minion
        password: |
          -----BEGIN PGP MESSAGE-----

          REDACTED
          -----END PGP MESSAGE-----
      database_uri: sqlite:///data/grainsrv.sqlite
      debug: false
      swagger: true

[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'grainsrv', OrderedDict([(u'server', OrderedDict([(u'ssl', OrderedDict([(u'enabled', False), (u'certfile', u'/ssl/cert.pem'), (u'keyfile', u'/ssl/key.pem')])), (u'container', OrderedDict([(u'image', u'myregistry.com/saltbox/grainsrv'), (u'tag', u'v0.2.0'), (u'mounts', OrderedDict([(u'data', u'/var/db/grainsrv'), (u'ssl', u'/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com')])), (u'publish', OrderedDict([(u'http', OrderedDict([(u'host', u'127.0.0.1'), (u'port', 5000)]))])), (u'env', OrderedDict([(u'VIRTUAL_HOST', u'salt-master-n01.dc1.mydomain.com')])), (u'restart', u'always')])), (u'settings', OrderedDict([(u'auth', OrderedDict([(u'enabled', True), (u'username', u'minion'), (u'password', u'-----BEGIN PGP MESSAGE-----\n\nhQEMA/dBgtQtEntJAQf/Z7KNj9qfO5+7PU+KoKBQvXcCE7gDWsocdTTQ4BQh4JXe\n3d0gVmbH7lIEzcYhdEyK+nw6vA+LDYmExIETN8RMamky/ouqd+Y8HMpX483xhWV4\nSTsQXfNfL+RbCtUOLW9AnlIszWgEk/gncGVwHCM2BReOR6CT//l2kid3OWx/jj0k\nl9dAKBLHnyTwyj995j5ZF3lD+TMSfCCHjaYuMV4xr1lwKODxyd7emz4+plWheP4Y\nQHc6wC0Hq3YOo5D9zvwURqHmF5nSkzS/DCq89AYvNXeGkIruYTrv4bz8lqarU6Ql\nwikDFvtfflc4cZClgtgMS1UWSsxT/5TTkcZpI7L4dtJLAda2a3D5eOZQlt6Sulfh\np9XmNH+ESlM6xBHMRsc8SUT5TVDAbjrVmtCYTGeSQXTOj+cJSrAraYbHKCXQrlo8\nlFi5e804gCam3mMc\n=IcK0\n-----END PGP MESSAGE-----\n')])), (u'database_uri', u'sqlite:///data/grainsrv.sqlite'), (u'debug', False), (u'swagger', True)]))]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/grainsrv/server.sls' using 'yaml' renderer: 0.00568604469299
[DEBUG   ] Reading GPG keys from: /etc/salt/gpgkeys
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/grainsrv/server.sls' using 'gpg' renderer: 0.0166430473328
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/app/nginx_proxy.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/master/git_pillar/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/nginx_proxy.sls' using 'jinja' renderer: 0.0083749294281
[DEBUG   ] Rendered data from file: /var/cache/salt/master/git_pillar/base/app/nginx_proxy.sls:
---

nginx_proxy:
  container:
    image: jwilder/nginx-proxy
    tag: alpine
    publish:
      http:
        host: 0.0.0.0
        port: 80
      https:
        host: 0.0.0.0
        port: 443
    restart: always
  settings:
    ipv6: false
    vhost_overrides_path: /opt/nginx/vhost.d
  ssl:
    enabled: true
  vhosts:
    - name: 'salt-master-n01.dc1.mydomain.com'
      ssl:
        enabled: true
        certfile: /etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/fullchain.pem
        keyfile: /etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/privkey.pem

[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'nginx_proxy', OrderedDict([(u'container', OrderedDict([(u'image', u'jwilder/nginx-proxy'), (u'tag', u'alpine'), (u'publish', OrderedDict([(u'http', OrderedDict([(u'host', u'0.0.0.0'), (u'port', 80)])), (u'https', OrderedDict([(u'host', u'0.0.0.0'), (u'port', 443)]))])), (u'restart', u'always')])), (u'settings', OrderedDict([(u'ipv6', False), (u'vhost_overrides_path', u'/opt/nginx/vhost.d')])), (u'ssl', OrderedDict([(u'enabled', True)])), (u'vhosts', [OrderedDict([(u'name', u'salt-master-n01.dc1.mydomain.com'), (u'ssl', OrderedDict([(u'enabled', True), (u'certfile', u'/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/fullchain.pem'), (u'keyfile', u'/etc/letsencrypt/live/salt-master-n01.dc1.mydomain.com/privkey.pem')]))])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/app/nginx_proxy.sls' using 'yaml' renderer: 0.00465416908264
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/init.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/master/git_pillar/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/init.sls' using 'jinja' renderer: 0.00559401512146
[DEBUG   ] Rendered data from file: /var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/init.sls:
---

include:
  - targeted.salt_master_n01_dc1_mydomain_com.users

[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'include', [u'targeted.salt_master_n01_dc1_mydomain_com.users'])])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/init.sls' using 'yaml' renderer: 0.00080680847168
[DEBUG   ] compile template: /var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/users.sls
[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'users', OrderedDict([(u'saltbot', OrderedDict([(u'fullname', u'Salt Telegram Bot'), (u'password', u'-----BEGIN PGP MESSAGE-----\n\nREDACTED\n-----END PGP MESSAGE-----\n'), (u'hash_password', True), (u'enforce_password', True), (u'home', u'/var/empty'), (u'createhome', False), (u'sudouser', False), (u'shell', u'/bin/false'), (u'manage_bashrc', False), (u'manage_profile', False), (u'manage_vimrc', False), (u'groups', [u'salt'])]))]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/users.sls' using 'yaml' renderer: 0.0036199092865
[DEBUG   ] Reading GPG keys from: /etc/salt/gpgkeys
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/git_pillar/base/targeted/salt_master_n01_dc1_mydomain.com/users.sls' using 'gpg' renderer: 0.01602602005
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pull.ipc
[TRACE   ] IPCServer: Handling connection to address: 
[DEBUG   ] Sending event: tag = minion/refresh/salt-master-n01.dc1.mydomain.com; data = {u'Minion data cache refresh': 'salt-master-n01.dc1.mydomain.com', u'_stamp': '2018-11-17T20:09:26.021244'}
[TRACE   ] get_event() received = {u'tag': 'minion/refresh/salt-master-n01.dc1.mydomain.com', u'data': {u'Minion data cache refresh': u'salt-master-n01.dc1.mydomain.com', u'_stamp': u'2018-11-17T20:09:26.021244'}}
[TRACE   ] get_event() received = {u'tag': 'minion/refresh/salt-master-n01.dc1.mydomain.com', u'data': {u'Minion data cache refresh': u'salt-master-n01.dc1.mydomain.com', u'_stamp': u'2018-11-17T20:09:26.021244'}}
[DEBUG   ] Gathering reactors for tag minion/refresh/salt-master-n01.dc1.mydomain.com
[TRACE   ] Master function call _pillar took 0.557945966721 seconds
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[TRACE   ] AES payload received with command _return
[DEBUG   ] LazyLoaded local_cache.prep_jid
[INFO    ] Got return from salt-master-n01.dc1.mydomain.com for job 20181117210925244993
[DEBUG   ] Sending event: tag = salt/job/20181117210925244993/ret/salt-master-n01.dc1.mydomain.com; data = {'fun_args': [], 'jid': '20181117210925244993', 'return': ['Pillar failed to render with the following messages:', "Rendering SLS 'app.grainsrv.server' failed. Please see master log for details."], 'retcode': 5, 'success': True, 'cmd': '_return', u'_stamp': '2018-11-17T20:09:26.072656', 'fun': 'state.highstate', 'id': 'salt-master-n01.dc1.mydomain.com', 'out': 'highstate'}
[TRACE   ] get_event() received = {u'tag': 'salt/job/20181117210925244993/ret/salt-master-n01.dc1.mydomain.com', u'data': {u'fun_args': [], u'jid': u'20181117210925244993', u'return': [u'Pillar failed to render with the following messages:', u"Rendering SLS 'app.grainsrv.server' failed. Please see master log for details."], u'retcode': 5, u'success': True, u'cmd': u'_return', u'_stamp': u'2018-11-17T20:09:26.072656', u'fun': u'state.highstate', u'id': u'salt-master-n01.dc1.mydomain.com', u'out': u'highstate'}}
[TRACE   ] get_event() received = {u'tag': 'salt/job/20181117210925244993/ret/salt-master-n01.dc1.mydomain.com', u'data': {u'fun_args': [], u'jid': u'20181117210925244993', u'return': [u'Pillar failed to render with the following messages:', u"Rendering SLS 'app.grainsrv.server' failed. Please see master log for details."], u'retcode': 5, u'success': True, u'cmd': u'_return', u'_stamp': u'2018-11-17T20:09:26.072656', u'fun': u'state.highstate', u'id': u'salt-master-n01.dc1.mydomain.com', u'out': u'highstate'}}
[DEBUG   ] Gathering reactors for tag salt/job/20181117210925244993/ret/salt-master-n01.dc1.mydomain.com
[TRACE   ] Master function call _return took 0.00589394569397 seconds
[TRACE   ] Process manager iteration
[TRACE   ] Process manager iteration
[DEBUG   ] git_pillar remote 'master https://myvcs.com/saltbox/pillars.git' is up-to-date
[DEBUG   ] pygit2 does not support detecting stale refs for authenticated remotes, saltenvs will not reflect branches/tags removed from remote 'master https://myvcs.com/saltbox/pillars.git'
[DEBUG   ] Removed update lock for git_pillar remote 'master https://myvcs.com/saltbox/pillars.git'
[TRACE   ] ==== evaluating schedule now None =====
[DEBUG   ] This salt-master instance has accepted 6 minion keys.
## ^C: salt-master shutting down after this point ##

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version: Salt: 2018.3.2

Dependency Versions: cffi: 1.9.1 cherrypy: unknown # (installed via pip: CherryPy==17.0.0) dateutil: Not Installed docker-py: 3.5.0 gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.9.4 libgit2: 0.27.4 libnacl: Not Installed M2Crypto: 0.24.0 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: Not Installed pycparser: 2.18 pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: 0.27.1 Python: 2.7.13 (default, Sep 26 2018, 18:42:22) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 16.0.2 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.4.3 ZMQ: 4.2.1

System Versions: dist: debian 9.6 locale: UTF-8 machine: x86_64 release: 4.9.0-8-amd64 system: Linux version: debian 9.6

garethgreenaway commented 6 years ago

@pirogoeth Thanks for the report. Do you see the same results if you use salt commands instead of using pepper? Since pepper is using the Salt API, do you see any errors in the Salt API logs?

pirogoeth commented 6 years ago

@garethgreenaway That's actually one thing I noticed after some more troubleshooting -- a salt-call state.highstate on the machine would not fail, but pepper would "fail". Unfortunately it looks like the issue has gone away (!?) now, as a state.highstate call from pepper completes successfully now.

garethgreenaway commented 6 years ago

@pirogoeth thanks for the update. Sounds like maybe a disconnect or other issue connecting to the salt API. Perhaps it had run out of available connections.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.