search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.11k stars 5.47k forks source link

file.managed with S3 Source errors out with obscure message #38836

Closed toanctruong closed 7 years ago

toanctruong commented 7 years ago

Description of Issue/Question

I have a simple state to download a binary file into my home directory from S3. I have an IAM role for my EC2 instance that provides full access to the S3 bucket. Manually running salt-call s3.get mys3bucket /infrastructure/atlassian/atlassian-servicedesk-3.2.6-x64.bin works just fine. Tried Googling for Exception: 'FSClient' object has no attribute 'utils' and I couldn't find anything useful.

Setup

Currently running masterless with GitFS Backend 

[root@amznlinux gitfs]# cat /etc/salt/minion.d/0.conf
master: localhost
file_client: local
fileserver_backend:
  - git
gitfs_provider: gitpython
gitfs_remotes:
  - git@github.com:myprivaterepo/repo.git
gitfs_root: srv/salt
pillar_roots:
  base:
    - /srv/pillars/dev
minion_data_cache: False
jira_sd_binary:
  file.managed:
    - source: s3://mys3bucket/infrastructure/atlassian/atlassian-servicedesk-3.2.6-x64.bin
    - name: /home/ec2-user/atlassian-servicedesk-3.2.6-x64.bin
    - skip_verify: true
    - unless: ls /home/ec2-user/atlassian-servicedesk-3.2.6-x64.bin
    - order: 1

Steps to Reproduce Issue

[root@amznlinux gitfs]# salt-call state.highstate
[ERROR   ] Failed to cache s3://mys3bucket/infrastructure/atlassian/atlassian-servicedesk-3.2.6-x64.bin: Could not fetch from s3://mys3bucket/infrastructure/atlassian/atlassian-servicedesk-3.2.6-x64.bin. Exception: 'FSClient' object has no attribute 'utils'

local:
----------
          ID: jira_sd_binary
    Function: file.managed
        Name: /home/ec2-user/atlassian-servicedesk-3.2.6-x64.bin
      Result: False
     Comment: Failed to cache s3://mys3bucket/infrastructure/atlassian/atlassian-servicedesk-3.2.6-x64.bin: Could not fetch from s3://mys3bucket/infrastructure/atlassian/atlassian-servicedesk-3.2.6-x64.bin. Exception: 'FSClient' object has no attribute 'utils'
     Started: 00:34:33.112108
    Duration: 10.059 ms
     Changes:

Versions Report

[root@amznlinux tmp]# salt-call --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.1
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Sep  1 2016, 22:14:00)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist:
        machine: x86_64
        release: 4.4.30-32.54.amzn1.x86_64
         system: Linux
        version: Not Installed
gtmanfred commented 7 years ago

Could you run this with salt-call and -l debug and then provide us with the full stack trace?

Thanks, Daniel

toanctruong commented 7 years ago

Here is the -l debug output.

[root@teamcitymaster opt]# salt-call state.apply teamcity_master.deploy -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/0.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/0.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: teamcitymaster.prod.corp-prod.uswest2.i-0c9fe65859c6aedc3
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/0.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/0.conf
[DEBUG   ] Please install 'virt-what' to improve results of the 'virtual' grain.
[DEBUG   ] Determining pillar cache
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] gitpython gitfs_provider enabled
[DEBUG   ] LazyLoaded git.envs
[DEBUG   ] Updating git fileserver cache
[DEBUG   ] Set update lock for gitfs remote 'git@github.com:myprivaterepo/repo.git'
[DEBUG   ] Fetching gitfs remote 'git@github.com:myprivaterepo/repo.git'
[DEBUG   ] Popen(['git', 'fetch', '-v', 'origin'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=True, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch-check'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] gitfs has updated 'origin/master' for remote 'git@github.com:myprivaterepo/repo.git' from eb4b4ca to e43ef23
[DEBUG   ] Popen(['git', 'remote', 'prune', '--dry-run', 'origin'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Removed update lock for gitfs remote 'git@github.com:myprivaterepo/repo.git'
[DEBUG   ] Popen(['git', 'cat-file', '--batch'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch-check'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Determining pillar cache
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Popen(['git', 'cat-file', '--batch-check'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch-check'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] In saltenv 'base', looking at rel_path 'teamcity_master/deploy.sls' to resolve 'salt://teamcity_master/deploy.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/teamcity_master/deploy.sls' to resolve 'salt://teamcity_master/deploy.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://teamcity_master/deploy.sls'
[DEBUG   ] No dest file found
[DEBUG   ] Popen(['git', 'cat-file', '--batch-check'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch-check'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'cat-file', '--batch'], cwd=/var/cache/salt/minion/gitfs/1b812f6cdfc47681580c61dce1b797e29b64f8d7be20b8184f8391a391de3756, universal_newlines=False, shell=None)
[INFO    ] Fetching file from saltenv 'base', ** done ** 'teamcity_master/deploy.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/teamcity_master/deploy.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/teamcity_master/deploy.sls' using 'jinja' renderer: 0.00268220901489
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/teamcity_master/deploy.sls:
teamcity_get_pkg:
  file.managed:
    - source: s3://my-private-s3-bucket/infrastructure/teamcity/TeamCity-10.0.4.tar.gz
    - name: /home/ec2-user/TeamCity-10.0.4.tar.gz
    - unless: ls /home/ec2-user/TeamCity-10.0.4.tar.gz
    - skip_verify: True
#
# teamcity_extract_pkg:
#   archive.extracted:
#     - name: /opt
#     - source: /home/ec2-user/TeamCity-10.0.4.tar.gz
#     - user: teamcity
#     - group: root
#
# /opt/TeamCity/conf/server.xml:
#   file.managed:
#     - name: /opt/TeamCity/conf/server.xml
#     - source: salt://teamcity_master/files/server.xml
#     - user: teamcity
#     - group: root
#     - mode: 644
#
# /opt/TeamCity/bin/teamcity-server.sh:
#   file.managed:
#     - name: /opt/TeamCity/bin/teamcity-server.sh
#     - source: salt://teamcity_master/files/teamcity-server.sh
#     - user: teamcity
#     - group: root
#     - mode: 644
#
# /opt/TeamCity/buildAgent/bin/agent.sh:
#   file.managed:
#     - name: /opt/TeamCity/buildAgent/bin/agent.sh
#     - source: salt://teamcity_master/files/agent.sh
#     - user: teamcity
#     - group: root
#     - mode: 644
#
# /etc/init.d/teamcity:
#   file.managed:
#     - name: /etc/init.d/teamcity
#     - source: salt://teamcity_master/files/teamcity
#     - user: root
#     - group: root
#     - mode: 644

[DEBUG   ] LazyLoaded config.get
[DEBUG   ] Results of YAML rendering:
OrderedDict([('teamcity_get_pkg', OrderedDict([('file.managed', [OrderedDict([('source', 's3://my-private-s3-bucket/infrastructure/teamcity/TeamCity-10.0.4.tar.gz')]), OrderedDict([('name', '/home/ec2-user/TeamCity-10.0.4.tar.gz')]), OrderedDict([('unless', 'ls /home/ec2-user/TeamCity-10.0.4.tar.gz')]), OrderedDict([('skip_verify', True)])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/teamcity_master/deploy.sls' using 'yaml' renderer: 0.00557613372803
[DEBUG   ] LazyLoaded file.managed
[INFO    ] Running state [/home/ec2-user/TeamCity-10.0.4.tar.gz] at time 23:05:25.830467
[INFO    ] Executing state file.managed for /home/ec2-user/TeamCity-10.0.4.tar.gz
[DEBUG   ] LazyLoaded cmd.has_exec
[DEBUG   ] LazyLoaded boto.assign_funcs
[DEBUG   ] SMBIOS: neither dmidecode nor smbios found!
[DEBUG   ] Could not LazyLoad augeas.execute: 'augeas.execute' is not available.
[DEBUG   ] LazyLoaded zabbix.host_create
[DEBUG   ] LazyLoaded splunk.list_users
[DEBUG   ] LazyLoaded x509.get_pem_entry
[DEBUG   ] LazyLoaded win_servermanager.install
[DEBUG   ] Could not LazyLoad firewall.get_config: 'firewall.get_config' is not available.
[DEBUG   ] LazyLoaded victorops.create_event
[DEBUG   ] LazyLoaded openvswitch.bridge_create
[DEBUG   ] LazyLoaded nxos.cmd
[DEBUG   ] LazyLoaded reg.read_value
[DEBUG   ] LazyLoaded boto_cloudtrail.exists
[DEBUG   ] LazyLoaded apache.a2ensite
[DEBUG   ] LazyLoaded mongodb.user_exists
[DEBUG   ] LazyLoaded zabbix.usergroup_create
[DEBUG   ] Could not LazyLoad pkg.get_selections: 'pkg.get_selections' is not available.
[DEBUG   ] LazyLoaded postgres.datadir_init
[DEBUG   ] LazyLoaded postgres.tablespace_exists
[DEBUG   ] LazyLoaded cisconso.set_data_value
[DEBUG   ] LazyLoaded boto_lambda.function_exists
[DEBUG   ] LazyLoaded zfs.create
[DEBUG   ] LazyLoaded postgres.cluster_exists
[DEBUG   ] LazyLoaded keystone.auth
[DEBUG   ] LazyLoaded postgres.language_create
[DEBUG   ] Could not LazyLoad redis.set_key: 'redis.set_key' is not available.
[DEBUG   ] LazyLoaded boto_cognitoidentity.describe_identity_pools
[DEBUG   ] LazyLoaded pecl.list
[DEBUG   ] LazyLoaded ifttt.trigger_event
[DEBUG   ] LazyLoaded tls.cert_info
[DEBUG   ] LazyLoaded boto_apigateway.describe_apis
[DEBUG   ] LazyLoaded memcached.status
[DEBUG   ] LazyLoaded apache.a2enmod
[DEBUG   ] LazyLoaded lvs.get_rules
[DEBUG   ] LazyLoaded boto_asg.exists
[DEBUG   ] LazyLoaded apache.a2enconf
[DEBUG   ] Could not LazyLoad influxdb.db_exists: 'influxdb.db_exists' is not available.
[DEBUG   ] LazyLoaded zabbix.user_create
[DEBUG   ] LazyLoaded rdp.enable
[DEBUG   ] LazyLoaded postgres.user_exists
[DEBUG   ] LazyLoaded snapper.diff
[DEBUG   ] LazyLoaded tomcat.status
[DEBUG   ] LazyLoaded boto_datapipeline.create_pipeline
[DEBUG   ] LazyLoaded boto_rds.exists
[DEBUG   ] LazyLoaded boto_vpc.exists
[DEBUG   ] LazyLoaded win_snmp.get_agent_settings
[DEBUG   ] LazyLoaded bower.list
[DEBUG   ] LazyLoaded monit.summary
[DEBUG   ] Could not LazyLoad docker.version: 'docker.version' is not available.
[DEBUG   ] LazyLoaded eselect.exec_action
[DEBUG   ] LazyLoaded ipset.version
[DEBUG   ] LazyLoaded apache.config
[DEBUG   ] LazyLoaded selinux.getenforce
[DEBUG   ] LazyLoaded npm.list
[DEBUG   ] LazyLoaded boto_iot.policy_exists
[DEBUG   ] LazyLoaded dockerng.version
[DEBUG   ] LazyLoaded win_dns_client.add_dns
[DEBUG   ] LazyLoaded splunk_search.get
[DEBUG   ] LazyLoaded portage_config.get_missing_flags
[DEBUG   ] LazyLoaded boto_elasticsearch_domain.exists
[DEBUG   ] LazyLoaded chocolatey.install
[DEBUG   ] Could not LazyLoad vmadm.create: 'vmadm.create' is not available.
[DEBUG   ] Registered VCS backend: git
[DEBUG   ] Registered VCS backend: hg
[DEBUG   ] Registered VCS backend: svn
[DEBUG   ] Registered VCS backend: bzr
[DEBUG   ] LazyLoaded ps.pkill
[DEBUG   ] LazyLoaded stormpath.create_account
[DEBUG   ] LazyLoaded chassis.cmd
[DEBUG   ] LazyLoaded win_path.rehash
[DEBUG   ] LazyLoaded postgres.create_extension
[DEBUG   ] Could not LazyLoad influxdb.db_exists: 'influxdb.db_exists' is not available.
[DEBUG   ] Could not LazyLoad lgpo.set: 'lgpo.set' is not available.
[DEBUG   ] LazyLoaded nftables.version
[DEBUG   ] LazyLoaded xmpp.send_msg
[DEBUG   ] LazyLoaded boto_asg.exists
[DEBUG   ] LazyLoaded zpool.create
[DEBUG   ] LazyLoaded github.list_users
[DEBUG   ] LazyLoaded postgres.privileges_grant
[DEBUG   ] LazyLoaded zk_concurrency.lock
[DEBUG   ] Could not LazyLoad influxdb08.db_exists: 'influxdb08.db_exists' is not available.
[DEBUG   ] LazyLoaded esxi.cmd
[DEBUG   ] LazyLoaded win_dacl.add_ace
[DEBUG   ] LazyLoaded cyg.list
[DEBUG   ] LazyLoaded virt.node_info
[DEBUG   ] LazyLoaded ddns.update
[DEBUG   ] LazyLoaded win_iis.create_site
[DEBUG   ] LazyLoaded acme.cert
[DEBUG   ] LazyLoaded boto_cloudwatch_event.exists
[DEBUG   ] LazyLoaded trafficserver.set_config
[DEBUG   ] LazyLoaded win_smtp_server.get_server_setting
[DEBUG   ] LazyLoaded lvs.get_rules
[DEBUG   ] Could not LazyLoad influxdb08.db_exists: 'influxdb08.db_exists' is not available.
[DEBUG   ] LazyLoaded postgres.user_exists
[DEBUG   ] LazyLoaded boto_s3_bucket.exists
[DEBUG   ] LazyLoaded postgres.group_create
[DEBUG   ] LazyLoaded postgres.schema_exists
[DEBUG   ] LazyLoaded kapacitor.version
[DEBUG   ] LazyLoaded layman.add
[DEBUG   ] LazyLoaded chef.client
[DEBUG   ] LazyLoaded openvswitch.port_add
[DEBUG   ] LazyLoaded makeconf.get_var
[DEBUG   ] LazyLoaded glusterfs.list_volumes
[DEBUG   ] LazyLoaded win_pki.get_stores
[DEBUG   ] LazyLoaded zabbix.hostgroup_create
[INFO    ] Executing command ['git', '--version'] in directory '/root'
[DEBUG   ] stdout: git version 2.7.4
[DEBUG   ] Could not LazyLoad elasticsearch.exists: 'elasticsearch.exists' is not available.
[DEBUG   ] Could not LazyLoad file.mod_run_check: 'file.mod_run_check' is not available.
[INFO    ] Executing command 'ls /home/ec2-user/TeamCity-10.0.4.tar.gz' in directory '/root'
[DEBUG   ] output: ls: cannot access /home/ec2-user/TeamCity-10.0.4.tar.gz: No such file or directory
[DEBUG   ] Last command return code: 2
[DEBUG   ] LazyLoaded git.envs
[ERROR   ] Failed to cache s3://my-private-s3-bucket/infrastructure/teamcity/TeamCity-10.0.4.tar.gz: Could not fetch from s3://my-private-s3-bucket/infrastructure/teamcity/TeamCity-10.0.4.tar.gz. Exception: 'FSClient' object has no attribute 'utils'
[INFO    ] Completed state [/home/ec2-user/TeamCity-10.0.4.tar.gz] at time 23:05:26.404981 duration_in_ms=574.515
[DEBUG   ] File /var/cache/salt/minion/accumulator/139653390002960 does not exist, no need to cleanup.
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: teamcity_get_pkg
    Function: file.managed
        Name: /home/ec2-user/TeamCity-10.0.4.tar.gz
      Result: False
     Comment: Failed to cache s3://my-private-s3-bucket/infrastructure/teamcity/TeamCity-10.0.4.tar.gz: Could not fetch from s3://my-private-s3-bucket/infrastructure/teamcity/TeamCity-10.0.4.tar.gz. Exception: 'FSClient' object has no attribute 'utils'
     Started: 23:05:25.830466
    Duration: 574.515 ms
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time: 574.515 ms

For completeness, since this is a different node but exhibiting the same issue.

[root@teamcitymaster opt]# cat /etc/salt/minion.d/0.conf
master: localhost
file_client: local
fileserver_backend:
  - git
gitfs_provider: gitpython
gitfs_remotes:
  - git@github.com:myprivaterepo/repo.git
gitfs_root: srv/salt
pillar_roots:
  base:
    - /srv/pillars/dev
minion_data_cache: False
[root@teamcitymaster opt]# salt-call --versions
Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.1
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Sep  1 2016, 22:14:00)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist:
        machine: x86_64
        release: 4.4.30-32.54.amzn1.x86_64
         system: Linux
        version: Not Installed
rsalmond commented 7 years ago

@gtmanfred I just spent some time digging into this and have found that at this point for some reason the self.utils object does not exist despite giving the appearance of being created at instantiation.

Inserting the line self.utils = salt.loader.utils(self.opts) directly above line 516 in fileclient.py corrects the problem but I doubt that is an acceptable fix.

rsalmond commented 7 years ago

Oh also, this can be easily reproduced with:

sudo salt-call --local cp.get_url s3://whatever/doesntmatter

mattgrahamdevops commented 7 years ago

Thats odd. Im getting the same problem on a packer build that is using salt masterless. I was just googling when I came across this post. Pretty sure it was working not that long ago for me too.

MasterNayru commented 7 years ago

Is the problem not that on line 1344 of fileclient.py that you need a line like self.utils = salt.loader.utils(self.opts)? Or maybe something along the lines of Client.__init__(self, opts) ? Line 517 of the same file will explode without self.utils being set somehow.

rsalmond commented 7 years ago

@MasterNayru yeah I tried adding an explicit call to RemoteClient.__init__ in FSClient too, didn't work as smoothly as I'd hoped.

# salt-call --local cp.get_url s3://foo/bar
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'master_uri'
Traceback (most recent call last):
  File "/tmp/.venv/bin/salt-call", line 11, in <module>
    sys.exit(salt_call())
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/scripts.py", line 379, in salt_call
    client.run()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/cli/caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/modules/cp.py", line 320, in get_url
    result = _client().get_url(path, dest, makedirs, saltenv)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/modules/cp.py", line 103, in _client
    _mk_client()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/modules/cp.py", line 96, in _mk_client
    salt.fileclient.get_file_client(__opts__)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/fileclient.py", line 57, in get_file_client
    }.get(client, RemoteClient)(opts)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/fileclient.py", line 1360, in __init__
    RemoteClient.__init__(self, opts)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/fileclient.py", line 968, in __init__
    self.channel = salt.transport.Channel.factory(self.opts)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/transport/__init__.py", line 49, in factory
    return ReqChannel.factory(opts, **kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/transport/client.py", line 25, in factory
    sync = SyncWrapper(AsyncReqChannel.factory, (opts,), kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/utils/async.py", line 60, in __init__
    self.async = method(*args, **kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/transport/client.py", line 108, in factory
    return salt.transport.zeromq.AsyncZeroMQReqChannel(opts, **kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/transport/zeromq.py", line 85, in __new__
    new_obj.__singleton_init__(opts, **kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/transport/zeromq.py", line 153, in __singleton_init__
    self.auth = salt.crypt.AsyncAuth(self.opts, io_loop=self._io_loop)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/crypt.py", line 335, in __new__
    key = cls.__key(opts)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/crypt.py", line 352, in __key
    opts['master_uri'],  # master ID
KeyError: 'master_uri'

@mattgrahamdevops yeah this appears to be a regression.

$ sudo salt-call --local cp.get_url s3://whateven/thing

Passed invalid arguments: get_url() takes at least 2 arguments (1 given).

Usage:

    Used to get a single file from a URL.

    The default behaviour is to write the fetched file to the given
    destination path. To simply return the text contents instead, set destination to
    None.

    CLI Example:

    .. code-block:: bash

        salt '*' cp.get_url salt://my/file /tmp/mine
        salt '*' cp.get_url http://www.slashdot.org /tmp/index.html

$ salt --version
salt 2016.3.1 (Boron)

---

# salt-call --local cp.get_url s3://foo/bar
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
MinionError: Could not fetch from s3://foo/bar. Exception: 'FSClient' object has no attribute 'utils'
Traceback (most recent call last):
  File "/tmp/.venv/bin/salt-call", line 11, in <module>
    sys.exit(salt_call())
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/scripts.py", line 379, in salt_call
    client.run()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/cli/caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/modules/cp.py", line 320, in get_url
    result = _client().get_url(path, dest, makedirs, saltenv)
  File "/tmp/.venv/local/lib/python2.7/site-packages/salt/fileclient.py", line 530, in get_url
    'Could not fetch from {0}. Exception: {1}'.format(url, exc)
MinionError: Could not fetch from s3://foo/bar. Exception: 'FSClient' object has no attribute 'utils'

# salt --version
salt 2016.11.3 (Carbon)
gtmanfred commented 7 years ago

Can you check if this fixes your issue?

https://github.com/saltstack/salt/pull/39609

It should initialize the utils and serial stuff like remoteclient and localclient do, and allow for using the s3:// as the source of files.

Thanks, Daniel

rsalmond commented 7 years ago

@gtmanfred I will make some time later today to try that if you like but the stack trace in this comment was the result when I attempted to add that change locally.

gtmanfred commented 7 years ago

That appears to be when you added RemoteClient, which does other things, Client just initializes self.serial and self.utils

My PR only intializes Client.

rsalmond commented 7 years ago

@gtmanfred Ah! Thanks for pointing that out, I missed that. I will try your approach.

rsalmond commented 7 years ago

@gtmanfred confirmed this eliminates the 'FSClient' object has no attribute 'utils' exception in my environment. Thanks for your quick work!