Closed msciciel closed 8 years ago
rallytime
commented
9 years ago @msciciel Hmm...Any other information you can provide? You say it only happens occasionally and on different boxes each time? Any similarities between these boxes? And only when you run state.highstate? Any other configs you've changed?
@basepi or @jfindlay or @cachedout Any of you heard of anything similar to this lately?
jfindlay
commented
9 years ago I think this is a duplicate of #20777. See also, https://github.com/saltstack/salt/issues/18729#issuecomment-66640876.
msciciel
commented
9 years ago It looks random. I have check of last highstate run and it raports this error. I do not run highstate manually. If check report that error occured and then i login into machine and run highstate then it's always ok. It's happens always on few boxes from 1600.
Is this related to pillar data or maybe formulas ? I have pillar data updated from git repo with 5 minutes interval.
mimianddaniel
commented
9 years ago Didnt see this issue until upgraded to RC2. Happens everytime for us using salt
Traceback (most recent call last):
File "/opt/blue-python/2.7/lib/python2.7/site-packages/salt/master.py", line 1415, in run_func
ret = getattr(self, func)(load)
File "/opt/blue-python/2.7/lib/python2.7/site-packages/salt/master.py", line 1270, in _syndic_return
self._return(ret)
File "/opt/blue-python/2.7/lib/python2.7/site-packages/salt/master.py", line 1244, in _return
self.opts, load, event=self.event, mminion=self.mminion)
File "/opt/blue-python/2.7/lib/python2.7/site-packages/salt/utils/job.py", line 55, in store_job
load.update({'user': ret_['user']})
TypeError: string indices must be integers, not strAlso, salt-run jobs.lookup_jid doesnt return anything now but i can see the ret events are flowing in. for my issue, at least this commit looks suspicious: 7a8b7a2558acf9652e1ffdb8459afd664f3752a3
Salt: 2015.2.0
Python: 2.7.9 (default, Dec 30 2014, 13:07:54)
Jinja2: 2.7.3
M2Crypto: 0.22
msgpack-python: 0.4.2
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.11
ioflo: Not Installed
PyZMQ: 14.4.1
RAET: Not Installed
ZMQ: 4.0.5
Mako: Not Installed
bmcclure
commented
9 years ago I am getting these errors, too. It seems mostly random. I have been commenting out things from my highstate to try to determine what could be happening.
It does NOT occur if I run state.highstate on one single minion. I also have NOT seen it occur if I don't use any formulas. However, if I have any formulas active, and if I am calling state.highstate on two or more minions, I often (not always) see the error on the first one, or several, minions, and the last one, or several, minions completes successfully.
Data failed to compile:
----------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2853, in call_highstate
top = self.get_top()
File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2355, in get_top
tops = self.get_tops()
File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2228, in get_tops
saltenv
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 147, in cache_file
return self.get_url(path, '', True, saltenv)
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 522, in get_url
return self.get_file(url, dest, makedirs, saltenv)
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 992, in get_file
if not data['data']:
TypeError: string indices must be integers, not strVersions:
Salt: 2014.7.2
Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.0.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: 0.9.1
Debian source package: 2014.7.2+ds-1trusty2I am now seeing the same error when I run state.highstate with just one minion--it just seems less frequent.
cachedout
commented
9 years ago After reviewing the code in the about-to-be-released update to 2014.7, I believe this issue has been resolved. I am marking his as Fixed Pending Verification until somebody can verify a fix in the next release.
gcamus
commented
9 years ago I am getting the same error too on versions 2014.7.2 or 2015.0.2rc1 but i think i find a workaround to set the saltenv on every command even if for base.
I remark that the problem appears when GitFs and ext_pillar git are activated and some branches exist on repositories
For exemple : salt '*' state.highstate saltenv=base
msciciel
commented
9 years ago I have ext pillar disabled and error sometimes occurs.
wwentland
commented
9 years ago We are seeing this all the time ever since we upgraded to 2014.7.2, quite annoying
basepi
commented
9 years ago 2014.7.4 has been tagged and we're just waiting on packagers. If someone wants to install from source or pip in the meantime and see if this is resolved, that would be lovely. Otherwise we'll wait until the official announcement.
PitrJ
commented
9 years ago I am trying to solve this issue as well. I made a state that basically just did a file.manged and nothing more. From a client I did the following:
while [ $? == 0 ] ; do salt-call -l debug state.sls testing/pieter; doneAll went fine the first 14 times, after that it broke:
[DEBUG ] Fetching file from saltenv 'base', ** attempting ** 'salt://testing/pieter.sls'
[ERROR ] Data is
Passed invalid arguments: string indices must be integers, not str
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/salt/cli/caller.py", line 129, in call
ret['return'] = func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/salt/modules/state.py", line 461, in sls
high_, errors = st_.render_highstate({saltenv: mods})
File "/usr/lib/python2.6/site-packages/salt/state.py", line 2743, in render_highstate
sls, saltenv, mods, matches)
File "/usr/lib/python2.6/site-packages/salt/state.py", line 2426, in render_state
state_data = self.client.get_state(sls, saltenv)
File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 423, in get_state
dest = self.cache_file(path, saltenv)
File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 147, in cache_file
return self.get_url(path, '', True, saltenv)
File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 522, in get_url
return self.get_file(url, dest, makedirs, saltenv)
File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 992, in get_file
if not data['data']:
TypeError: string indices must be integers, not strHere is my very simple sls file:
/tmp/test.pieter:
file.managed:
- source: salt://testing/pieter/testfile
- user: root
- group: root
- mode: 644Forgot the versions:
Salt: 2014.7.2
Python: 2.6.6 (r266:84292, Oct 12 2012, 14:23:48)
Jinja2: unknown
M2Crypto: 0.20.2
msgpack-python: 0.1.9.final
msgpack-pure: Not Installed
pycrypto: 2.0.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.3.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: Not InstalledI think the problem is really due to GitFS and list of env issue. If you enable GitFS and you have multiple repos you have to create same branches on all your git repos or else use master conf (See below example) :
gitfs_env_blacklist:
- release*
gitfs_env_whitelist:
- base
- bambooIf you want you can check all declared GitFS envs names in file : /var/cache/salt/master/gitfs/envs.p
PitrJ
commented
9 years ago We are using 1 repo as gitfs_remote.
PitrJ
commented
9 years ago Running the same loop with saltenv=base did not solve the problem. Still broke after about 20 cycles.
rallytime
commented
9 years ago @PitrJ It looks like you're on 2014.7.2, where we had some gitfs regressions. These have now been fixed in 2014.7.4, which was recently tagged and is now in the packaging stage. I'd recommend giving that tag a try, if you're able. If not, please let us know if you still hit this issue once the packages for 2014.7.4 are available and you've had a chance to retest.
wwentland
commented
9 years ago @rallytime Which particular issue are you referring to and in which commit was it fixed? Given that .4 won't make it into the repositories for some time I might consider building and distributing a locally maintained version or fix it in whatever way is appropriate, but for that I'd need to know more details.
rallytime
commented
9 years ago @babilen Unfortunately, I don't have a particular commit or pull request number offhand. I was going off of @cachedout's comment above about reviewing the code and marking this as fixed pending verification.
I think really we are just wondering if someone can test this on 2014.7.4, where the tag (v2014.7.4) is pushed to the saltstack repo, or on packages once they come out. We'll certainly leave this open until we have confirmed that this was fixed, or revisit it if more work needs to be done.
wwentland
commented
9 years ago @rallytime That's perfectly fine. I might have just given it a try if it had boiled down to a simple patch that I can test easily. It's just that I haven't immediately seen something related while glimpsing over the git log and was curious.
MrMarvin
commented
9 years ago From the piece of code I looked at I believe that the issue is master side only. I upgraded our master to git v2014.7.4. It now runs
root@master2-salt1:~# salt-call --versions-report
Salt: 2014.7.4
Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.0.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: Not InstalledThe minions however are still running 2014.7.2 from ubuntu trusty repos.
For me the issue is still there.
And yes, our setup uses multiple gitpython based gitfs repos, some with a non-default branch. Just to make sure we are talking about the same error here.
andrew-vant
commented
9 years ago I have this on 2014.7.2 as well. It is intermittent, and the trace usually seems to get come from within a defaults.get call. (possibly pillar.get as well, I am unsure)
I will re-test when the next package is available.
rallytime
commented
9 years ago Thanks very much for testing this on a newer version @MrMarvin. I'll remove the fixed pending verification label.
MrMarvin
commented
9 years ago Actually... after upgrading both master and most of our minions to 2014.7.4 I do not see this issue any more. Be aware that is isn't reliable to reproduce though.
Anyone else with this finding?
wwentland
commented
9 years ago If somebody knows what exactly causes this I would be greatful as I would love to roll out a fix for .2 so that we can start using salt again in our infrastructure.
furbaez
commented
9 years ago Wait this was fixed or not?
luitzifa
commented
9 years ago We experience this bug too and we do NOT use gitfs. But it seems related to git.latest. While highstate we deploy same stuff for ~28 users in a row and then it just happens right after a git.latest state. And then every file.managed I made several runs, the user changes but its everytime after git.latest.
----------
ID: repo_typo3src_user12
Function: git.latest
Name: git://git.typo3.org/Packages/TYPO3.CMS.git
Result: True
Comment: Repository git://git.typo3.org/Packages/TYPO3.CMS.git cloned to /var/www/user12/resources/typo3_src
Started: 18:05:21.152685
Duration: 46022.282 ms
Changes:
----------
new:
git://git.typo3.org/Packages/TYPO3.CMS.git
revision:
b3201dd1e9ffd33dbd55912626d6de248e8d5696
Name: /var/www/user12/resources/typo3_src - Function: file.directory - Result: Clean
----------
ID: fpm_user12_service
Function: service.running
Name: fpm_user12
Result: True
Comment: Service fpm_user12 has been enabled, and is running
Started: 18:06:15.130102
Duration: 622.705 ms
Changes:
----------
fpm_user12:
True
----------
ID: /etc/php5/fpm_user13
Function: file.directory
Result: True
Comment: Directory /etc/php5/fpm_user13 updated
Started: 18:06:15.753276
Duration: 1.915 ms
Changes:
----------
/etc/php5/fpm_user13:
New Dir
----------
ID: /etc/php5/fpm_user13/fpm.conf
Function: file.managed
Result: True
Comment: File /etc/php5/fpm_user13/fpm.conf updated
Started: 18:06:15.782279
Duration: 22.394 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: /etc/php5/fpm_user13/php.ini
Function: file.managed
Result: True
Comment: File /etc/php5/fpm_user13/php.ini updated
Started: 18:06:15.812258
Duration: 119.29 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: /etc/init.d/fpm_user13
Function: file.managed
Result: True
Comment: File /etc/init.d/fpm_user13 updated
Started: 18:06:15.953115
Duration: 16.0 ms
Changes:
----------
diff:
New file
mode:
0755
----------
ID: resources_user13
Function: file.directory
Name: /var/www/user13/resources
Result: True
Comment: Directory /var/www/user13/resources updated
Started: 18:06:16.222539
Duration: 2.165 ms
Changes:
----------
/var/www/user13/resources:
New Dir
----------
ID: repo_Frontend_user13
Function: git.latest
Name: ssh://gerrit.mycompany.de/Soft1/Frontend
Result: True
Comment: Repository ssh://gerrit.mycompany.de/Soft1/Frontend cloned to /var/www/user13/git
Started: 18:06:16.245665
Duration: 90398.259 ms
Changes:
----------
new:
ssh://gerrit.mycompany.de/Soft1/Frontend
revision:
4d3007cf449a574bb254644a2bccc011484ffe0e
----------
ID: repo_typo3src_user13
Function: git.latest
Name: git://git.typo3.org/Packages/TYPO3.CMS.git
Result: True
Comment: Repository git://git.typo3.org/Packages/TYPO3.CMS.git cloned to /var/www/user13/resources/typo3_src
Started: 18:08:51.778060
Duration: 53832.769 ms
Changes:
----------
new:
git://git.typo3.org/Packages/TYPO3.CMS.git
revision:
b3201dd1e9ffd33dbd55912626d6de248e8d5696
Name: /var/www/user13/resources/typo3_src - Function: file.directory - Result: Clean
----------
ID: fpm_user13_service
Function: service.running
Name: fpm_user13
Result: True
Comment: Service fpm_user13 has been enabled, and is running
Started: 18:09:53.544995
Duration: 489.643 ms
Changes:
----------
fpm_user13:
True
----------
ID: /etc/php5/fpm_user14
Function: file.directory
Result: True
Comment: Directory /etc/php5/fpm_user14 updated
Started: 18:09:54.035073
Duration: 1.957 ms
Changes:
----------
/etc/php5/fpm_user14:
New Dir
----------
ID: /etc/php5/fpm_user14/fpm.conf
Function: file.managed
Result: False
Comment: Unable to manage file: string indices must be integers, not str
Started: 18:09:54.063658
Duration: 129.251 ms
Changes:
----------
ID: /etc/php5/fpm_user14/php.ini
Function: file.managed
Result: False
Comment: Unable to manage file: string indices must be integers, not str
Started: 18:09:54.205878
Duration: 7.74 ms
Changes:
----------> salt-minion --versions-report
Salt: 2014.7.2
Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
Jinja2: 2.6
M2Crypto: 0.21.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.6
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 13.1.0
RAET: Not Installed
ZMQ: 3.2.3
Mako: Not Installed
Debian source package: 2014.7.2+ds-1~bpo70+1I'd suggest to add the Fixed Pending Verification label again and wait for some other confirmation if this is fixed in 2014.7.4. My first negative findings were inaccurate and it may be in fact fixed.
clinta
commented
9 years ago If I could get some packages of 2014.7.4 in the salt testing ppa, I'd be happy to test. I see this issue every day. But I don't want to deploy pip throughout my environment to test the newer version.
joehealy
commented
9 years ago packages going to salt testing ppa now
clinta
commented
9 years ago Thanks. Been testing for a few hours as I develop and push states and haven't seen the issue since upgrading. I'll update if I see it again.
rallytime
commented
9 years ago Thanks again for the updates here everyone. We should be announcing 2014.7.4 officially here (likely today) and the packages will roll out from testing repos to stable in the coming days. I'll add the fixed-pending verification label as suggested, and if people do still see this again after upgrading, do let us know.
Thanks again!
andrew-vant
commented
9 years ago Datapoint: The 2014.7.4 package is now available from whatever repo I'm pointed to, and after upgrading both my master and my minions, I no longer have this problem.
wwentland
commented
9 years ago So far we haven't seen it either after upgrading to .4.
luitzifa
commented
9 years ago 2014.7.4 works fine, thank you very much!
Summary
---------------
Succeeded: 1650 (changed=1406)
Failed: 0
---------------
Total states run: 1650Awesome! Thanks for testing this everyone! With at least 4 confirmations that this has been fixed, I feel comfortable closing this issue. If this pops up again after upgrading to 2014.7.4, please leave a comment and we can re-address the issue.
romlok
commented
9 years ago I just encountered this error message as well, running under 2014.7.4. I believe the cause of my troubles was different to the original problem, but the issue title is the error message, so I'm posting my experience here for future reference.
TL;DR: If you see this error, check file permissions.
Running salt-call state.sls postgres.installed:
----------
ID: postgres-hba
Function: file.managed
Name: /etc/postgreql/9.1/main/pg_hba.conf
Result: False
Comment: Unable to manage file: string indices must be integers, not str
Started: 14:11:06.492990
Duration: 10.757 ms
Changes:
----------The state in question:
postgres-hba:
file.managed:
- name: /etc/postgreql/9.1/main/pg_hba.conf
- source: salt://postgres/pg_hba.conf.jinja
- template: jinjaRelevant snippet of salt-call output:
[INFO ] Running state [/etc/postgresql/9.1/main/pg_hba.conf] at time 14:28:14.608229
[INFO ] Executing state file.managed for /etc/postgresql/9.1/main/pg_hba.conf
[DEBUG ] Fetching file from saltenv 'base', ** attempting ** 'salt://postgres/pg_hba.conf.jinja'
[ERROR ] Data is
[DEBUG ] Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 1393, in managed
**kwargs
File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 2521, in get_managed
sfn = __salt__['cp.cache_file'](source, saltenv)
File "/usr/lib/python2.7/dist-packages/salt/modules/cp.py", line 337, in cache_file
result = __context__['cp.fileclient'].cache_file(path, saltenv)
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 147, in cache_file
return self.get_url(path, '', True, saltenv)
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 522, in get_url
return self.get_file(url, dest, makedirs, saltenv)
File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 993, in get_file
if not data['data']:
TypeError: string indices must be integers, not str
[ERROR ] Unable to manage file: string indices must be integers, not str
[INFO ] Completed state [/etc/postgresql/9.1/main/pg_hba.conf] at time 14:28:14.618628Versions:
# salt-call --versions-report
Salt: 2014.7.4
Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
Jinja2: 2.6
M2Crypto: 0.21.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.6
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 13.1.0
RAET: Not Installed
ZMQ: 3.2.3
Mako: 0.7.0
Debian source package: 2014.7.4+ds-1~bpo70+1However, when I removed template: jinja from the state, the error changed to:
Source file salt://postgres/pg_hba.conf.jinja not found
So, after close inspection of the filenames, and much copy & pasting of paths, I discovered that it was actually the file permissions of /srv/salt/postgres/pg_hba.conf.jinja which didn't allow the salt master access to the file...
So I guess the ultimate upshot here is that the core issue behind this ticket has not really been remedied, which is: Whatever is triggering this particular error message needs to be more careful about not masking other more useful error messages.
jfindlay
commented
9 years ago @raumkraut, thanks for the report. I'll be looking into this.
cachedout
commented
9 years ago I can reproduce this by establishing a ZMQ channel to the master and then bringing down the master and then bringing it back up. After doing so, all calls to the channel's send method will return an empty string, since the channel no longer has the right AES key. This is fixed in #23154
msciciel
commented
9 years ago Any progress with fix ?
rallytime
commented
9 years ago @msciciel Does the pull request mentioned above resolve this issue for you?
msciciel
commented
9 years ago Is 2014.7.5 or 2015.5.0 fixed version ?
rallytime
commented
9 years ago @msciciel It look like the fix didn't make it into an official release yet. The pull request was merged in 2015.5 right after we cut 2015.5.2. Therefore, the fix will be available when 2015.5.3 is released.
We can definitely keep this open until you've had a chance to test it.
msciciel
commented
9 years ago Ok. I'll verify after release of 2015.5.3. I saw today that problem still exists.
yermulnik
commented
9 years ago Similar error appeared in archive.extracted after upgrading Salt to 2015.5.3
ID: /usr/local/etc/salt/states/deploy/node_srvt/worker_oe/bin/1.6.0.3610.tmp
Function: archive.extracted
Result: False
Comment: Unable to manage file: string indices must be integers, not str
Started: 11:30:52.130212
Duration: 2330.451 ms
Changes:How can this be fixed? It breaks our deployment system! =(
rallytime
commented
9 years ago @yermulnik Do you have a similar stacktrace as the others in this thread? Or is this specifically only happening when using an archive.extracted state? I am just trying ascertain if this is the same bug or one more specific to the state you're running.
yermulnik
commented
9 years ago That's only what I get. No stacktrace. I've encountered it only when using archive.extracted.
rallytime
commented
9 years ago @yermulnik Does it happen each time you run the state, or only intermittently? If it's consistently failing each time, it might be a different bug. If not, then something is still amiss here.
yermulnik
commented
9 years ago Yes, it happens each time I run the state. And it appeared since I had upgraded to 2015.5.3 yesterday. I don't know if it's a different bug. I searched for the "string indices must be integers, not str" error and found 2 issues: this one and https://github.com/saltstack/salt/issues/21799 - so I decided to ask for help here and there. Should I create new issue to not confuse similar but not same issues?
rallytime
commented
9 years ago @yermulnik Would you mind opening a new one? That sounds different to me than these other ones and that way we can get some more eyes on it. Thanks!
Occasionaly during highstate i got error (it looks like it occur on random machine with random time):
Salt highstate error: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/salt/state.py", line 2834, in call_highstate top = self.get_top() File "/usr/lib/python2.6/site-packages/salt/state.py", line 2336, in get_top tops = self.get_tops() File "/usr/lib/python2.6/site-packages/salt/state.py", line 2209, in get_tops saltenv File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 147, in cache_file return self.get_url(path, '', True, saltenv) File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 521, in get_url return self.get_file(url, dest, makedirs, saltenv) File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 991, in get_file if not data['data']: TypeError: string indices must be integers, not strSalt: 2014.7.1 Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) Jinja2: 2.7.2 M2Crypto: 0.20.2 msgpack-python: 0.1.13 msgpack-pure: Not Installed pycrypto: 2.0.1 libnacl: Not Installed PyYAML: 3.10 ioflo: Not Installed PyZMQ: 14.3.1 RAET: Not Installed ZMQ: 4.0.4 Mako: Not InstalledAny ideas what is wrong ?