search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.1k stars 5.47k forks source link

[2016.3.0] TypeError: 'int' object has no attribute '__getitem__' #33524

Closed syphernl closed 8 years ago

syphernl commented 8 years ago

Description of Issue/Question

Running the following on the salt-master:

 salt-run state.event pretty=True

results in an exception:

[DEBUG   ] Sending event - data = {'jid': '20160526080133349560', 'return': 'Exception occurred in runner state.event: Traceback (most recent call last):\n  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 346, in low\n    data[\'return\'] = self.functions[fun](*args, **kwargs)\n  File "/usr/lib/python2.7/dist-packages/salt/runners/state.py", line 196, in event\n    node=node)\n  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 1658, in event\n    ret = sevent.get_event(full=True)\n  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 579, in get_event\n    ret = self._get_event(wait, tag, match_func, no_block)\n  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 484, in _get_event\n    raw = self.subscriber.read_sync(timeout=wait)\n  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 654, in read_sync\n    return ret_future.result()\n  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result\n    raise_exc_info(self._exc_info)\n  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run\n    yielded = self.gen.send(value)\n  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 631, in _read_sync\n    raise exc_to_raise  # pylint: disable=E0702\nTypeError: \'int\' object has no attribute \'__getitem__\'\n', 'success': False, '_stamp': '2016-05-26T06:01:33.674314', 'user': 'sudo_vagrant', 'fun': 'runner.state.event'}
[DEBUG   ] LazyLoaded local_cache.prep_jid
[INFO    ] Runner completed: 20160526080133349560
[DEBUG   ] Runner return: Exception occurred in runner state.event: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 346, in low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/runners/state.py", line 196, in event
    node=node)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 1658, in event
    ret = sevent.get_event(full=True)
  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 579, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 484, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 654, in read_sync
    return ret_future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 631, in _read_sync
    raise exc_to_raise  # pylint: disable=E0702
TypeError: 'int' object has no attribute '__getitem__'

Versions Report

Salt Version:
           Salt: 2016.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: 1.4.3
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-83-generic
         system: Linux
        version: Ubuntu 14.04 trusty
syphernl commented 8 years ago

This actually also happens with something simple like:

salt 'example.com' test.ping

Traceback:

Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 468, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/salt.py", line 158, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 622, in cmd_cli
    **kwargs):
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1294, in get_cli_event_returns
    **kwargs
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 913, in get_iter_returns
    for raw in ret_iter:
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 848, in get_returns_no_block
    raw = self.event.get_event(wait=0.01, tag=tag, match_type=match_type, full=True, no_block=True)
  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 579, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/lib/python2.7/dist-packages/salt/utils/event.py", line 484, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 654, in read_sync
    return ret_future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/salt/transport/ipc.py", line 631, in _read_sync
    raise exc_to_raise  # pylint: disable=E0702
TypeError: 'int' object has no attribute '__getitem__'
thatch45 commented 8 years ago

@syphernl can you send us your master config too? I am having a hard time reproducing this

syphernl commented 8 years ago

I tried with both my and the one provided in the Ubuntu package, but if it's configuration related it must be one of the master.d files then... Op 26 mei 2016 5:43 p.m. schreef Thomas S Hatch notifications@github.com:@syphernl can you send us your master config too? I am having a hard time reproducing this

—You are receiving this because you were mentioned.Reply to this email directly or view it on GitHub

thatch45 commented 8 years ago

so default configuration? ZeroMQ? Not TCP transport? Ok, I am still hunting

thatch45 commented 8 years ago

Also, does this fail consistently after restarting the master?

thatch45 commented 8 years ago

I ask about restarting the master because one of the logical problems that we see with the loader systems in salt is that since it is hot loading modules all the time it will oad the new plugin modules. 99.9% of the time this is good, but sometimes it can leave an upgraded and not restarted master in a bad state

syphernl commented 8 years ago

The package update restarts the master (or at least should)

This is my master config:

enable_gpu_grains: False
auto_accept: True
log_level: debug
log_level_logfile: info

And my master.d (concatenated):

file_recv: True

peer:
  .*:
    - network.ip_addrs

pillar_roots:
    base:
        - /srv/salt/environments/prod/pillars
    qa:
        - /srv/salt/environments/qa/pillars    
        - /srv/salt/environments/prod/pillars
    dev:
        - /srv/salt/environments/dev/pillars
        - /srv/salt/environments/qa/pillars
        - /srv/salt/environments/prod/pillars

pillar_opts: False

reactor:
  - 'backupninja/client/register':
    - /srv/salt/formulas/backupninja/backupninja/master/reactors/backup_client_register.sls

file_roots:
    base:
        - /srv/salt/environments/prod/states
        - /srv/salt/environments/prod/templates
        - /srv/salt/environments/prod/files
        - /srv/salt/environments/prod/formulas
    qa:
        - /srv/salt/environments/qa/states    
        - /srv/salt/environments/prod/states
        - /srv/salt/environments/qa/templates  
        - /srv/salt/environments/prod/templates
        - /srv/salt/environments/qa/files  
        - /srv/salt/environments/prod/files
        - /srv/salt/environments/qa/formulas
        - /srv/salt/environments/prod/formulas
    dev:
        - /srv/salt/environments/dev/states
        - /srv/salt/environments/qa/states
        - /srv/salt/environments/prod/states
        - /srv/salt/environments/dev/templates
        - /srv/salt/environments/qa/templates
        - /srv/salt/environments/prod/templates
        - /srv/salt/environments/dev/files
        - /srv/salt/environments/qa/files
        - /srv/salt/environments/prod/files        
        - /srv/salt/environments/dev/formulas
        - /srv/salt/environments/qa/formulas
        - /srv/salt/environments/prod/formulas

mine_functions:
  network.interfaces: []
Ch3LL commented 8 years ago

my understanding is that you would need to reboot the service after an upgrrade of the salt-master.

Maybe @thatch45 or @dmurphy18 can confirm this one for me though?

Also just to confirm @syphernl are you still seeing the error after the reboot?

syphernl commented 8 years ago

I haven't explicitly restarted the service since that is something the package (generally) does. That worked fine in the past every time. Op 27 mei 2016 7:33 p.m. schreef Megan Wilhite notifications@github.com:my understanding is that you would need to reboot the service after an upgrrade of the salt-master.

Maybe @thatch45 or @dmurphy18 can confirm this one for me though?

Also just to confirm @syphernl are you still seeing the error after the reboot?

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

cachedout commented 8 years ago

I can't reproduce this either. I can see the potential for this happening with a master that was upgraded and didn't restart though. We need @dmurphy18 to let us know if the restart should be happening if it still is.

syphernl commented 8 years ago

Even though there are processes running:

root@saltmaster:~# service salt-master stop
stop: Unknown instance:

After killing them all (killall salt-master) it properly showed stopped, then was able to restart and seems to be working fine now.

zhulh200868 commented 7 years ago

I have this problem:2017-01-05 15:18:16,721 [salt.transport.ipc][ERROR ][20270] Exception occurred while handling stream: 'int' object has no attribute 'getitem' Yesterday,I upgrade the salt to Salt: 2016.11.1 and can use the salt-master,but today it doesn't work.

xenophonf commented 6 years ago

This happened to me after a package upgrade on FreeBSD. Restarting the master corrected the issue.

Ch3LL commented 6 years ago

@zhulh200868 can you open another issue with more details related to your issue if its still an issue for you.

@xenophonf we don't build the freebsd pkgs but if you create an issue with the details including salt version we can ping the community member who manages that and get more eyes on your problem.