pkg.installed with sources doesn't work behind proxy

chrispetsos commented 5 years ago

Description of Issue/Question

When a minion runs behind a proxy, all pkg.installed states succeed, except those that define sources.

Setup

install_osquery:
  pkg.installed:
    - sources:
      - osquery: https://pkg.osquery.io/rpm/osquery-2.10.2-1.linux.x86_64.rpm

Steps to Reproduce Issue

Invoke the above state on the minion behind the proxy,

Logs:

[INFO    ] Running state [install_osquery] at time 15:50:07.614530
[INFO    ] Executing state pkg.installed for [install_osquery]
[DEBUG   ] Requesting URL https://pkg.osquery.io/rpm/osquery-2.10.2-1.linux.x86_64.rpm using GET method
[DEBUG   ] Using backend: requests
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1905, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1830, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/states/pkg.py", line 1548, in installed
    **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/states/pkg.py", line 716, in _find_install_targets
    cached_path = __salt__['cp.cache_file'](val, saltenv=kwargs['saltenv'])
  File "/usr/lib/python2.7/site-packages/salt/modules/cp.py", line 500, in cache_file
    result = _client().cache_file(path, saltenv, source_hash=source_hash)
  File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 193, in cache_file
    path, '', True, saltenv, cachedir=cachedir, source_hash=source_hash)
  File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 734, in get_url
    **get_kwargs
  File "/usr/lib/python2.7/site-packages/salt/utils/http.py", line 349, in query
    method, url, params=params, data=data, **req_kwargs
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))

Versions Report

$ salt --versions-report
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.10.0
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.0
         Python: 2.7.14 (default, Jan 31 2018, 02:12:13)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.10 Final
         locale: UTF-8
        machine: x86_64
        release: 2.6.32-696.16.1.el6.x86_64
         system: Linux
        version: CentOS 6.10 Final

I export the https_proxy, http_proxy and no_proxy env vars in file /etc/profile.d/proxy.sh. Also, using the requests backend. Observing the logs of our proxy server, other requests that are going out to the internet appear normally. The above, problematic request never reaches our proxy server.

garethgreenaway commented 5 years ago

@chrispetsos Thanks for the report.

mattLLVW commented 5 years ago

my two cents: are you able to download the package manually from the minion?

chrispetsos commented 5 years ago

Yes, I can successfully download the package manually using e.g. wget.

terminalmage commented 5 years ago

If you run salt someminion cp.cache_file https://pkg.osquery.io/rpm/osquery-2.10.2-1.linux.x86_64.rpm, do you get the same error?

terminalmage commented 5 years ago

Obviously, replace someminion with the minion ID of that same box behind the proxy.

chrispetsos commented 5 years ago

Sorry for the late response... yes, I get the same error with the cp.cache_file invocation too.

terminalmage commented 5 years ago

OK, and how is the proxy configured in Salt? Did you configure proxy_host, proxy_port, and optionally proxy_username and proxy_password?

https://docs.saltstack.com/en/latest/ref/configuration/minion.html#proxy-host

chrispetsos commented 5 years ago

This is a CentOS 6.10 minion I am talking about. What we've been doing so far and has worked for everything else except the pkg.installed states that define sources is: 1) Create a file in /etc/profile.d/ file which exports the http_proxy, https_proxy env variables to the correct host/port. It also exports the no_proxy env variable for targets which the minion doesn't need a proxy to connect to. 2) Configure the salt minion to use the requests backend. 3) Some yum-related setup for proxies in /etc/yum.conf.

Apart from this, setting the proxy_host, proxy_port in /etc/salt/minion doesn't seem to solve the problem. Invoking the cp.cache_file line you mentioned earlier, I receive the same error.

reinaldo-gomes commented 5 years ago

I'm facing the same issue on CentOS 7. I've configured http_proxy on /etc/yum.conf, /etc/environment, and proxy_host on /etc/salt/minion, but pkg.installed still ignores all proxy configuration when I pass "sources" arg.

dgengtek commented 4 years ago

I found I had to explicitly set http_proxy and https_proxy environment variables for the salt-minion services in the respective service files of the service manager since salt does not read any host environment files.

KyleMaas commented 7 months ago

I just ran into this as well.

saltstack / salt