search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.11k stars 5.47k forks source link

salt-ssh causing Artifactory failure, base64 dump #43641

Closed perfecto25 closed 5 years ago

perfecto25 commented 7 years ago

Description of Issue/Question

Running Salt-ssh command to query our boxes and get Splunk version, Salt-ssh caused our production Artifactory to error out during its next puppet run

Could not get latest version: Could not list gems: Execution of '/usr/bin/gem list --remote ^java$' returned 1: WARNING: RubyGems 1.2+ index not found for: ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError) (/Stage[main]/Common::Software/Package[java]/ensure) change from 0.3.00.2.0 to latest failed: Could not get latest version: Could not list gems: Execution of '/usr/bin/gem list --remote ^java$' returned 1: WARNING: RubyGems 1.2+ index not found for: (/Stage[main]/Common::Software/Package[java]/ensure) ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)

Salt-ssh dumps a base64 python dump which breaks Artifactory and also causes some alerts in our network intrusion monitors.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

Salt Version: Salt: 2016.11.5

Dependency Versions: cffi: 1.10.0 cherrypy: Not Installed dateutil: 2.6.0 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.9.6 libgit2: Not Installed libnacl: Not Installed M2Crypto: 0.21.1 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: Not Installed pycparser: 2.18 pycrypto: 2.6.1 pycryptodome: 3.4.3 pygit2: Not Installed Python: 2.7.5 (default, Nov 6 2016, 00:28:07) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 14.3.1 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 3.2.5

System Versions: dist: centos 7.3.1611 Core machine: x86_64 release: 3.10.0-514.16.1.el7.x86_64 system: Linux version: CentOS Linux 7.3.1611 Core

Logs on our Artifactory server

from /var/log/messages

Sep 19 16:48:49 artifactory.local ICAgIGlmIG5vdCBPUFRJT05TLnR0eToKICAgICAgICBzeXMuc3RkZXJyLndyaXRlKE9QVElPTlMu Sep 19 16:48:49 artifactory.local ZGVsaW1pdGVyICsgJ1xuJykKICAgICAgICBzeXMuc3RkZXJyLmZsdXNoKCkKICAgIGlmIE9QVElP Sep 19 16:48:49 artifactory.local TlMuY21kX3VtYXNrIGlzIG5vdCBOb25lOgogICAgICAgIG9sZF91bWFzayA9IG9zLnVtYXNrKE9Q Sep 19 16:48:49 artifactory.local VElPTlMuY21kX3VtYXNrKQogICAgaWYgT1BUSU9OUy50dHk6CiAgICAgICAgc3Rkb3V0LCBfID0g Sep 19 16:48:49 artifactory.local c3VicHJvY2Vzcy5Qb3BlbihzYWx0X2FyZ3YsIHN0ZG91dD1zdWJwcm9jZXNzLlBJUEUsIHN0ZGVy Sep 19 16:48:49 artifactory.local cj1zdWJwcm9jZXNzLlBJUEUpLmNvbW11bmljYXRlKCkKICAgICAgICBzeXMuc3Rkb3V0LndyaXRl Sep 19 16:48:49 artifactory.local KHN0ZG91dCkKICAgICAgICBzeXMuc3Rkb3V0LmZsdXNoKCkKICAgICAgICBpZiBPUFRJT05TLndp Sep 19 16:48:49 artifactory.local cGU6CiAgICAgICAgICAgIHNodXRpbC5ybXRyZWUoT1BUSU9OUy5zYWx0ZGlyKQogICAgZWxpZiBP Sep 19 16:48:49 artifactory.local UFRJT05TLndpcGU6CiAgICAgICAgc3VicHJvY2Vzcy5jYWxsKHNhbHRfYXJndikKICAgICAgICBz Sep 19 16:48:49 artifactory.local aHV0aWwucm10cmVlKE9QVElPTlMuc2FsdGRpcikKICAgIGVsc2U6CiAgICAgICAgc3Vi Sep 19 16:48:49 artifactory.local cy5jYWxsKHNhbHRfYXJndikKICAgIGlmIE9QVElPTlMuY21kX3VtYXNrIGlzIG5vdCBOb25lOgog Sep 19 16:48:49 artifactory.local ICAgICAgIG9zLnVtYXNrKG9sZF91bWFzaykKCmlmIF9fbmFtZV9fID09ICdfX21haW5fXyc6CiAg Sep 19 16:48:49 artifactory.local ICBzeXMuZXhpdChtYWluKHN5cy5hcmd2KSkK Sep 19 16:48:49 artifactory.local """).decode("utf-8")) Sep 19 16:48:54 artifactory.local exec(base64.b64decode("""IyBweWxpbnQ6IGRpc2FibGU9Vzk5MDMKJycnClRoaXMgaXMgYSBzaGltIHRoYXQgaGFuZGxlcyBj Sep 19 16:48:54 artifactory.local aGVja2luZyBhbmQgdXBkYXRpbmcgc2FsdCB0aGluIGFuZAp0aGVuIGludm9raW5nIHRoaW4uCgpU Sep 19 16:48:54 artifactory.local aGlzIGlzIG5vdCBpbnRlbmRlZCB0byBiZSBpbnN0YW50aWF0ZWQgYXMgYSBtb2R1bGUsIHJhdGhl Sep 19 16:48:54 artifactory.local ciBpdCBpcyBhCmhlbHBlciBzY3JpcHQgdXNlZCBieSBzYWx0LmNsaWVudC5zc2guU2luZ2xlLiAg Sep 19 16:48:54 artifactory.local SXQgaXMgaGVyZSwgaW4gYQpzZXBhcmF0ZSBmaWxlLCBmb3IgY29udmVuaWVuY2Ugb2YgZGV2ZWxv Sep 19 16:48:54 artifactory.local cG1lbnQuCicnJwoKZnJvbSBfX2Z1dHVyZV9fIGltcG9ydCBhYnNvbHV0ZV9pbXBvcnQKCmltcG9y Sep 19 16:48:54 artifactory.local dCBoYXNobGliCmltcG9ydCB0YXJmaWxlCmltcG9ydCBzaHV0aWwKaW1wb3J0IHN5cwppbXBvcnQg Sep 19 16:48:54 artifactory.local b3MKaW1wb3J0IHN0YXQKaW1wb3J0IHN1YnByb2Nlc3MKClRISU5fQVJDSElWRSA9ICdzYWx0LXRo Sep 19 16:48:54 artifactory.local aW4udGd6JwpFWFRfQVJDSElWRSA9ICdzYWx0LWV4dF9tb2RzLnRneicKCiMgS2VlcCB0aGVzZSBp Sep 19 16:48:54 artifactory.local biBzeW5jIHdpdGggc2FsdC9kZWZhdWx0cy9leGl0Y29kZXMucHkKRVhfVEhJTl9ERVBMT1kgPSAx Sep 19 16:48:54 artifactory.local MQpFWF9USElOX0NIRUNLU1VNID0gMTIKRVhfTU9EX0RFUExPWSA9IDEzCkVYX1NDUF9OT1RfRk9V Sep 19 16:48:54 artifactory.local TkQgPSAxNApFWF9DQU5UQ1JFQVQgPSA3MwoKCmNsYXNzIE

artifactory.local puppet-agent[2954]: Finished catalog run in 9.10 seconds Sep 19 17:36:50 artifactory.local puppet-agent[6275]: Finished catalog run in 8.56 seconds Sep 19 18:06:50 artifactory.local puppet-agent[9619]: Finished catalog run in 8.43 seconds Sep 19 18:36:53 artifactory.local puppet-agent[12942]: Could not get latest version: Could not list gems: Execution of '/usr/bin/gem list --remote ^lantern$' returned 1: WARNING: RubyGems 1.2+ index not found for:

from artifactory server /var/log/secure Sep 19 16:52:56 artifactory.local sshd[1775]: User child is on pid 1777 Sep 19 16:52:56 artifactory.local sudo: joe.user: TTY=unknown ; PWD=/home/joe.user ; USER=root ; COMMAND=/usr/bin/python2.6 -c import

Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) base64; Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) 9CSihvYmplY3QpOgogICAgIiIiQW4g Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) NWRiNjgwX3NhbHQnCk9QVElPTlMu Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) QoT1BUSU9OUy5zYWx0ZGlyKQog Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) IG5vdCByb290LCBiZSBjZXJ0 Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) ZvciBjaHVuayBpbiBpdGVy Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) ICAgICAgICAncnVubmlu Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) 5jKToKICAgICAgICAg Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) bl9wYXRoKSBvciBu Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) NlIGl0CgogICAg Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) X2V4dCgpCiAg Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) 5kIHN0ZGVy Sep 19 16:52:56 artifactory.local sudo: joe.user : (command continued) KQogICAg

gtmanfred commented 7 years ago

Unfortunately, We do not really know what artifactory is, can you provide us more information about how this is breaking the system?

We dump the whole set of python code that is going to be executed via salt-ssh as a base64 encoded blob.

https://github.com/saltstack/salt/blob/2017.7/salt/client/ssh/__init__.py#L1060

Than that is used to call salt-call on the salt-ssh minion, and the data is returned through the ssh process.

Thanks, Daniel

perfecto25 commented 7 years ago

artifactory is a java-based binary hosting server built by jFrog. This may be more to do with 'gem --list' command than Artifactory,

Execution of '/usr/bin/gem list --remote ^java$' returned 1: WARNING: RubyGems 1.2+ index not found for: ERROR: While executing gem ...

This happened right after I ran the salt-ssh with its base64 blob.

gtmanfred commented 7 years ago

What commands did you run through salt-ssh?

perfecto25 commented 7 years ago

was basic rpm -qa command

salt-ssh -i target cmd.run "rpm -qa | grep splunk"

perfecto25 commented 7 years ago

I will try to replicate the issue on a test environment.

gtmanfred commented 7 years ago

If you use regular salt-minion/salt-master setup, do you see the same issue happen?

This is odd, because salt-ssh runs in a self contained directory, and just by running the command, shouldn't cause a problem.

If you use pkg.version instead of using cmd.run do you see the same issue?

Thanks, Daniel

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.