search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.1k stars 5.47k forks source link

Unable to manage file: string indices must be integers, not str #21799

Closed syphernl closed 9 years ago

syphernl commented 9 years ago

Once in a while we see this error coming by whenever we run highstates. Its not clear where it originates from and what causes it (yet).

It may be related to the fact that our states/files are being git pull'ed every 5 minutes (via a cron, not via gitfs), except that the times do not correlate with the optional update and that it generally also happens even if nothing has changed in the repo.

Our minions are on a schedule to highstate every 15 minutes, which depends on when they were last highstated to spread the load a bit better.

A few minutes ago we had a bunch of minions highstating at more or less the same time, resulting in a flood of errors coming in.

This results in whole states breaking whenever file.managed is involved (direct or as a requisite):

* file.managed: Unable to manage file: string indices must be integers, not str (dnsmasq_cnames)
* file.managed: Unable to manage file: string indices must be integers, not str (dnsmasq_conf)
* service.running: One or more requisite failed: {\'dnsmasq.dnsmasq_hosts\': \'Unable to manage file: string indices must be integers, not str\', \'dnsmasq.dnsmasq_cnames\': \'Unable to manage file: string indices must be integers, not str\', \'dnsmasq.dnsmasq_conf_dir\': \'Recurse failed: none of the specified sources were found\', \'dnsmasq.dnsmasq_conf\': \'Unable to manage file: string indices must be integers, not str\'} (dnsmasq)
* file.managed: Unable to manage file: string indices must be integers, not str (/etc/resolv.conf)
* file.recurse: Recurse failed: none of the specified sources were found (dnsmasq_conf_dir)
* file.managed: Unable to manage file: string indices must be integers, not str (dnsmasq_hosts)

The same appears to be affecting the ssh_auth.present where the internal command to cp.fileclient results in traceback: 

* ssh_auth.present: An exception occurred in this state: Traceback (most recent call last):\n  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1531, in call\n    **cdata['kwargs'])\n  File "/usr/lib/python2.7/dist-packages/salt/states/ssh_auth.py", line 174, in present\n    saltenv=__env__)\n  File "/usr/lib/python2.7/dist-packages/salt/modules/cp.py", line 294, in get_file_str\n    fn_ = cache_file(path, saltenv)\n  File "/usr/lib/python2.7/dist-packages/salt/modules/cp.py", line 337, in cache_file\n    result = __context__['cp.fileclient'].cache_file(path, saltenv)\n  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 147, in cache_file\n    return self.get_url(path, '', True, saltenv)\n  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 522, in get_url\n    return self.get_file(url, dest, makedirs, saltenv)\n  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 992, in get_file\n    if not data['data']:\nTypeError: string indices must be integers, not str\n (the_username)

Version details:

                  Salt: 2014.7.2
                Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
 Debian source package: 2014.7.2+ds-1trusty2
``
rallytime commented 9 years ago

@syphernl I think you might be hitting the same thing as reported with ssh.auth_present as is reported in #21783 as they're seeing the same stacktrace on the same version on the same command.

However, i am not sure what is happening with the file.managed errors. Is there anything additional with your set-up that we should know (multi-master, syndic, masterless, etc?) Is it the same states that produce the errors each time, or different states that all use file.managed? (Looking for an example of what might be causing that, i.e. if there are any args in your state that might be causing the problem or something.)

syphernl commented 9 years ago

The ssh.auth_present error is different though, it could be related.

Regarding file.managed, our setup is just a single master. When errors occur, they hit the same files (for those that share states). Its generally the same files every time.

For instance, at 14-03 08:18 we had one and the same server was getting the same errors yesterday at 12:18 again.

In this case its our dnsmasq recursor which state (matching both times) is:

dnsmasq_cnames:
  file.managed:
    - name: {{ dnsmasq.dnsmasq_cnames }}
    - source: {{ salt['pillar.get']('dnsmasq:dnsmasq_cnames', 'salt://dnsmasq/files/dnsmasq.cnames') }}
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: dnsmasq
    - watch_in:
      - service: dnsmasq
rallytime commented 9 years ago

@basepi Have you seen this before? Or maybe @cachedout?

basepi commented 9 years ago

Probably a duplicate of #21480

Likely what we're expecting to be a dictionary is an error message (as a string) or similar. We need to get some logging in place to discover what the error message is -- I bet at that point the underlying problem becomes more clear.

variia commented 9 years ago

Hi,

I have also been experiencing this relatively frequently especially on RHEL6 during highstate run only.

$ salt --versions Salt: 2014.7.1 Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) Jinja2: 2.2.1 M2Crypto: 0.20.2 msgpack-python: 0.4.4 msgpack-pure: Not Installed pycrypto: 2.0.1 libnacl: Not Installed PyYAML: 3.10 ioflo: Not Installed PyZMQ: 14.3.1 RAET: Not Installed ZMQ: 3.2.5 Mako: Not Installed

Not as much on RHEL7 but I don't have a lot of those boxes so have less chance to trigger this bug...

Here is one of my most recent highstate runs:

      ID: cherrypy
Function: pkg.installed
    Name: python-cherrypy
  Result: True
 Comment: The following packages were installed/updated: python-cherrypy.
 Started: 03:38:36.938506
Duration: 17724.375 ms
 Changes:
          ----------
          python-cherrypy:
              ----------
              new:
                  3.2.2-3.el6
              old:
      ID: pip-install
Function: pkg.latest
    Name: python-pip
  Result: True
 Comment: Package python-pip is already up-to-date.
 Started: 03:38:54.670748
Duration: 3958.49 ms
 Changes:
      ID: halite
Function: pip.installed
    Name: halite == 0.1.17
  Result: True
 Comment: Package was successfully installed
 Started: 03:38:58.638326
Duration: 711.892 ms
 Changes:
          ----------
          halite==0.1.17:
              Installed
      ID: salt-master
Function: service.running
  Result: True
 Comment: Service restarted
 Started: 03:38:59.448079
Duration: 690.964 ms
 Changes:
          ----------
          salt-master:
              True
      ID: salt-api
Function: pkg.installed
  Result: True
 Comment: The following packages were installed/updated: salt-api.
 Started: 03:39:00.151727
Duration: 16384.477 ms
 Changes:
          ----------
          salt-api:
              ----------
              new:
                  2014.7.1-1.el6
              old:
      ID: salt-api
Function: service.running
  Result: True
 Comment: Service restarted
 Started: 03:39:16.608750
Duration: 3323.018 ms
 Changes:
          ----------
          salt-api:
              True
      ID: salt-ssh
Function: pkg.installed
  Result: True
 Comment: The following packages were installed/updated: salt-ssh.
 Started: 03:39:19.941961
Duration: 16349.771 ms
 Changes:
          ----------
          salt-ssh:
              ----------
              new:
                  2014.7.1-1.el6
              old:
      ID: salt-ssh
Function: file.managed
    Name: /etc/salt/roster
  Result: False
 Comment: Unable to manage file: string indices must be integers, not str
 Started: 03:39:36.298003
Duration: 7.168 ms
 Changes:
      ID: salt-vimfiles
Function: file.recurse
    Name: /usr/share/vim/vimfiles
  Result: False
 Comment: Recurse failed: none of the specified sources were found
 Started: 03:39:36.305391
Duration: 3.175 ms
 Changes:
      ID: pyflakes-vimfiles
Function: file.recurse
    Name: /usr/share/vim/vimfiles
  Result: False
 Comment: Recurse failed: none of the specified sources were found
 Started: 03:39:36.308767
Duration: 3.202 ms
 Changes:
      ID: nerdtree-vimfiles
Function: file.recurse
    Name: /usr/share/vim/vimfiles
  Result: False
 Comment: Recurse failed: none of the specified sources were found
 Started: 03:39:36.312169
Duration: 3.009 ms
 Changes:
      ID: awscli-install
Function: pip.installed
    Name: awscli
  Result: True
 Comment: Python package awscli already installed
 Started: 03:39:36.319005
Duration: 113.705 ms
 Changes:

What I find interesting is that I also have "none of the specified sources were found" errors too, which is rubbish, all my source files are correct and available.

Rerunning the highstate succeded with all 320 states the ones that fail tend to be the ones that require no change aka 'up-to-date' in my experience.

cachedout commented 9 years ago

This is a duplicate of #21480 which I have just fixed in #23154

rallytime commented 9 years ago

If this is a duplicate, then I am inclined to close this in favor of the other one that was filed before this one. What do you think @syphernl? Also, would you like to give the pull request mentioned above a try?

syphernl commented 9 years ago

I haven't noticed any issues lately (with 2014.7.4 / 2014.7.5). Is this a master or a minion fix?

DmitryKuzmenko commented 9 years ago

@syphernl today I worked on the issue investigation. It llooks @cachedout right, it should be fiexed by #23154

Could you please try to use 2014.7 git branch? The fix is there.

basepi commented 9 years ago

Is this a master or a minion fix?

Minion, I think. But you should upgrade both.

tomasfejfar commented 9 years ago

Seeing this with:

root@salt:~# salt-run --versions-report
                  Salt: 2014.7.5
                Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
                Jinja2: 2.6
              M2Crypto: 0.21.1
        msgpack-python: 0.1.10
          msgpack-pure: Not Installed
              pycrypto: 2.6
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 13.1.0
                  RAET: Not Installed
                   ZMQ: 3.2.3
                  Mako: 0.7.0
 Debian source package: 2014.7.5+ds-1~bpo70+1

And all minions up to date (i.e. same)

rallytime commented 9 years ago

Has anyone tried this on a newer version of salt (current stable is 2015.5.2) and can confirm that this is fixed?

yermulnik commented 9 years ago

Similar error appeared in archive.extracted after upgrading Salt to 2015.5.3 https://github.com/saltstack/salt/issues/21480

rallytime commented 9 years ago

Since we didn't hear back from about this lately and @DmitryKuzmenko has confirmed the fix from @cachedout, I am going to close this. I have asked @yermulnik to file a new issue with the details about his bug. Thanks everyone!