[FEATURE REQUEST] Cannot configure TLS parameters for consul.py module

[pgporada]

Description I am running a development consul cluster with mutual TLS configured. I want to use saltstack to help me implement/use the ACL system. From my testing, it doesn't appear that the consul module supports connecting to a TLS enabled consul cluster. I have explicitly disabled HTTP access.

Setup

$ salt-call consul.agent_self consul_url="https://127.0.1.1:8501"
local:
    ----------
    data:
        ----------
        error:
            HTTP 403: Forbidden
        status:
            403
    res:
        True

Expected behavior Specifically I need to be able to use the following flags:

consul acl list \
  -http-addr https://127.0.1.1:8501 \
  -tls-server-name client.cluster.consul \
  -ca-file /tmp/my-ca-cert.crt
  -client-cert /etc/consul/client.cert \
  -client-key /etc/consul/client.key

Versions Report

Salt Version:
          Salt: 3002.7

Dependency Versions:
          cffi: 1.14.0
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.10 (default, Sep 28 2021, 16:10:42)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-1049-kvm
        system: Linux
       version: Ubuntu 20.04 focal

Additionally, I think there's a bug in consul.acl_list. This specific field shouldn't be required, the documentation doesn't even mention it for this function. https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.consul.html#salt.modules.consul.acl_list

$ salt-call consul.acl_list consul_url="https://127.0.1.1:8501"
local:
    ----------
    message:
        Required parameter "id" is missing.
    res:
        False

[aaomidi]

So it seems this function here https://github.com/saltstack/salt/blob/fe3ee16a5610af05a7a533968f41f8df898ca345/salt/modules/consul.py#L43 is responsible for reaching out to consul. This function is called by all the exported functions of the consul module. So we'd need to modify the signature of all the functions to support taking a TLS CA and private key argument.

[dmurphy18]

Checking to see if PR https://github.com/saltstack/salt/pull/58101 affects this issue, feeling it will not.

[dmurphy18]

@pgporada What version of consul are you using ?, perhaps a cleaned up 'pip3 list' might be useful to determine what your system is using, in helping to resolve the issue

[dmurphy18]

It appears that this is really a feature request for the Salt consol.py module to support https and allow for the use of certs etc. Given Consol v1.5+ (current is v1.11), certain functionally requires the use of certs, a prime example of this is Kubernates support https://www.consul.io/docs/security/acl/auth-methods/kubernetes, where the CACert field is required.

The current implementation in Salt consol.py module for query using salt.utils.http.query does not even allow for the use of anything but relies on the functions default for port which is 80, and no cabundle is passed, for example:

  82     result = salt.utils.http.query(
  83         url,
  84         method=method,
  85         params=query_params,
  86         data=data,
  87         decode=True,
  88         status=True,
  89         header_dict=headers,
  90         opts=__opts__,
  91     )

From the Consul pages: https://www.consul.io/commands/rtt#consul_http_ssl_verify

The following environment variables control accessing the HTTP server via SSL:

CONSUL_HTTP_SSL Set this to enable SSL
CONSUL_HTTP_SSL_VERIFY Set this to disable certificate checking (not recommended)

Note: from the Consul documentation it appears that Consul is either HTTP or HTTPS but not both at the same time, implying a large switch to go from one to the other. But given the current movement away from use of HTTP to HTTPS, the need for this is only going to become greater as time passes.

Marked this as a Feature Request.

[pgporada]

@dmurphy18 I've got consul 1.11.2 and nomad 1.2.4 in my environment. The nomad module would also need this support.

[dmurphy18]

@pgporada Thanks for letting us know, will need to test with and without nomad for https support.

Note: there is currently no module for nomad in Salt, unsure if that will be required to implement the changes for the consul module, hopefully not and any usage will be purely between Hashicorp's Consul and Nomad.