[BUG]sdb:// entries in salt-cloud providers files not being resolved

[denz7201]

Description I have several entries such as "user: sdb://services/service_account" in my /etc/salt/cloud.providers.d/ files example:

Austin:
  driver: vmware
  user: sdb://mysqlite/svc_acct
  password: sdb://mysqlite/svc_pass
  url: 'vcenter-host'

Using the latest version salt-cloud I am getting errors. This works with the 2019.x versions. If I replace the sdb:// entries with the credentials all works properly. This works properly:

 salt-run sdb.get 'sdb://mysqlite/svc_acct'
 salt-run sdb.get 'sdb://mysqlite/svc_pass'

I have already tried the patch mentioned in #57016 to no avail

Steps to Reproduce the behavior `salt-cloud -f list_nodes Datacenter -ldebug [DEBUG ] Reading configuration from /etc/salt/cloud [DEBUG ] Reading configuration from /etc/salt/master [DEBUG ] Including configuration from '/etc/salt/master.d/external_pillar.conf' [DEBUG ] Reading configuration from /etc/salt/master.d/external_pillar.conf [DEBUG ] Including configuration from '/etc/salt/master.d/fileserver.conf' [DEBUG ] Reading configuration from /etc/salt/master.d/fileserver.conf [DEBUG ] Including configuration from '/etc/salt/master.d/reactor.conf' [DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf [DEBUG ] Including configuration from '/etc/salt/master.d/sdb.conf' [DEBUG ] Reading configuration from /etc/salt/master.d/sdb.conf [DEBUG ] Including configuration from '/etc/salt/master.d/spacewalk.conf' [DEBUG ] Reading configuration from /etc/salt/master.d/spacewalk.conf [DEBUG ] Changed git to gitfs in master opts' fileserver_backend list [DEBUG ] Using cached minion ID from /etc/salt/minion_id: centos8salttest [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] Missing configuration file: /etc/salt/cloud.providers [DEBUG ] Including configuration from '/etc/salt/cloud.providers.d/Austin.conf' [DEBUG ] Reading configuration from /etc/salt/cloud.providers.d/Austin.conf [DEBUG ] Including configuration from '/etc/salt/cloud.providers.d/US-saltify.conf' [DEBUG ] Reading configuration from /etc/salt/cloud.providers.d/US-saltify.conf [DEBUG ] Missing configuration file: /etc/salt/cloud.profiles [DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/us-salt-profile.conf' [DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/us-salt-profile.conf [DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/vmware-sizes.conf' [DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/vmware-sizes.conf [DEBUG ] Including configuration from '/etc/salt/cloud.profiles.d/vmware-tests.conf' [DEBUG ] Reading configuration from /etc/salt/cloud.profiles.d/vmware-tests.conf [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] LazyLoaded sqlite3.get [DEBUG ] Configuration file path: /etc/salt/cloud [WARNING ] Insecure logging configuration detected! Sensitive data may be logged. [INFO ] salt-cloud starting [DEBUG ] Marking 'base64_encode' as a jinja filter [DEBUG ] Marking 'base64_decode' as a jinja filter [DEBUG ] Marking 'md5' as a jinja filter [DEBUG ] Marking 'sha1' as a jinja filter [DEBUG ] Marking 'sha256' as a jinja filter [DEBUG ] Marking 'sha512' as a jinja filter [DEBUG ] Marking 'hmac' as a jinja filter [DEBUG ] Marking 'hmac_compute' as a jinja filter [DEBUG ] Marking 'random_hash' as a jinja filter [DEBUG ] Marking 'rand_str' as a jinja filter [DEBUG ] Marking 'file_hashsum' as a jinja filter [DEBUG ] Marking 'http_query' as a jinja filter [DEBUG ] Marking 'strftime' as a jinja filter [DEBUG ] Marking 'date_format' as a jinja filter [DEBUG ] Marking 'yaml_dquote' as a jinja filter [DEBUG ] Marking 'yaml_squote' as a jinja filter [DEBUG ] Marking 'yaml_encode' as a jinja filter [DEBUG ] Marking 'raise' as a jinja global [DEBUG ] Marking 'match' as a jinja test [DEBUG ] Marking 'equalto' as a jinja test [DEBUG ] Marking 'skip' as a jinja filter [DEBUG ] Marking 'sequence' as a jinja filter [DEBUG ] Marking 'to_bool' as a jinja filter [DEBUG ] Marking 'tojson' as a jinja filter [DEBUG ] Marking 'quote' as a jinja filter [DEBUG ] Marking 'regex_escape' as a jinja filter [DEBUG ] Marking 'regex_search' as a jinja filter [DEBUG ] Marking 'regex_match' as a jinja filter [DEBUG ] Marking 'regex_replace' as a jinja filter [DEBUG ] Marking 'uuid' as a jinja filter [DEBUG ] Marking 'unique' as a jinja filter [DEBUG ] Marking 'min' as a jinja filter [DEBUG ] Marking 'max' as a jinja filter [DEBUG ] Marking 'avg' as a jinja filter [DEBUG ] Marking 'union' as a jinja filter [DEBUG ] Marking 'intersect' as a jinja filter [DEBUG ] Marking 'difference' as a jinja filter [DEBUG ] Marking 'symmetric_difference' as a jinja filter [DEBUG ] Could not LazyLoad parallels.avail_sizes: 'parallels' virtual returned False [DEBUG ] LazyLoaded parallels.avail_locations [DEBUG ] LazyLoaded proxmox.avail_sizes [DEBUG ] Trying to execute 'vmware.list_nodes' with the following kwargs: {} [ERROR ] There was an error running the function: Could not connect to host '10.212.18.151'. Please check the debug log for more information. Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 268, in _get_service_instance mechanism=mechanism) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 847, in SmartConnect sslContext) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 726, in FindSupportedVersion httpProxyPort) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 643, in GetServiceVersionDescription httpProxyHost, httpProxyPort) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 608, in __GetElementTree conn.request("GET", path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 773, in init self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 292, in _get_service_instance mechanism=mechanism) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 867, in SmartConnect mechanism=mechanism) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 266, in Connect keyFile, certFile, thumbprint, sslContext, connectionPoolTimeout) File "/usr/local/lib/python3.6/site-packages/pyVim/connect.py", line 390, in __Login x = content.sessionManager.Login(user, pwd, None) File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 511, in _InvokeMethod list(map(CheckField, info.params, args)) File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 1098, in CheckField % (info.name, info.type.name, valType.name)) TypeError: For "userName" expected type str, but got bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/cloud/cli.py", line 269, in run self.function_provider, self.function_name, kwargs File "/usr/lib/python3.6/site-packages/salt/cloud/init.py", line 1585, in do_function driver: self.cloudsfun File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 1727, in list_nodes vm_list = salt.utils.vmware.get_mors_with_properties(_get_si(), vim.VirtualMachine, vm_properties) File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 261, in _get_si port=port) File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 428, in get_service_instance domain) File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 322, in _get_service_instance raise salt.exceptions.VMwareConnectionError(err_msg) salt.exceptions.VMwareConnectionError: Could not connect to host 'datacenter'. Please check the debug log for more information. `

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ``` Salt Version: Salt: 3000.2 Dependency Versions: cffi: 1.11.5 cherrypy: Not Installed dateutil: 2.6.1 docker-py: Not Installed gitdb: 2.0.3 gitpython: 2.1.11 Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.6.2 mysql-python: Not Installed pycparser: 2.14 pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Nov 21 2019, 19:31:34) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 17.0.0 smmap: 2.0.3 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: centos 8.1.1911 Core locale: UTF-8 machine: x86_64 release: 4.18.0-80.11.2.el8_0.x86_64 system: Linux version: CentOS Linux 8.1.1911 Core ```

Additional context Add any other context about the problem here.

[waynew]

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)

It looks like the SSL cert isn't valid, or at least whatever machine was trying to connect to '10.212.18.151'. was not happy about it.

Are you sure that your SSL cert is valid?

[denz7201]

Changing the sdb:// entry out for actual credentials results in correct operation. That IP is the vcenter machine

[denz7201]

BTW it is pretty typical in a private datacenter for VMWare to be using snakeoil certificates.

[cingeyedog]

@denz7201 We had the same issue using the sqllite3 version of sdb.

TypeError: For "userName" expected type str, but got bytes

We switched over to using yaml for our SDB. salt.sdb.yaml module

The other advantage of this was that we could use the GPG encrypt the username / password and now can store the SDB in git repos without exposing username / password.

[root@salt-master]# cat /etc/salt/master.d/vmware_yaml.conf
vcenter:
  driver: yaml
  files:
    - /etc/salt/cloud.providers.d/vcenter.yaml
  gpg: True
[root@salt-master]# cat /etc/salt/cloud.providers.d/vcenter.yaml
prod_url: PROD_URL.example.com
dr_url: DR_URL.example.com
user: |
  -----BEGIN PGP MESSAGE-----
  Version: GnuPG v2.0.22 (GNU/Linux)
  hQIMA+BNIwZHcIj9AQ//arLioYZp9muvQIV/O1Gfts5aW7l0VtfNpe2oO6b0cY/n
  e/KpDDns4WxyxNGdNiE1FYg+JzaaEVQDXIA5a3lGDjXyma8rVRJftKnn3Dc+MCnV
  sSNezyisPsLwIWd8BgvydScvfpxOGV/bvMXEY9M3dhYPlnvMUGgg1fpf+2FlH1dJ
  bUIG7tLBx3RhaBHKZ0EkQVJQuxoN7Rq22gsH4jeAP6K1Um2xEF6CnYW+FrChP71/
  2gX2ljuD7xBdhwnYSECcGDYsB0SWSERKSRUirO5WP4RIIiv/gYZjh9i9wjgirYpc
  g2WJl8Nn1vIGA6dsLplzYH+/20aC+9BMBSE7j9cXeTPUMeray4xs6Fj1di7lfOal
  iD9uXMK3Q2UKhhkHYE6SL+fH0aMz/mxYyZme8kUiXus4oaFMoMCCf82qLClDDXas
  DMSHNXNnxlkaEUX9IbShzTyheyRkpZOq1GVPk0K84YRKtsEbioGRsPcx2Y6CXMdo
  SscRGtZRiKev1r05rDZmFw5TNdAecw6l+HfvDE+Aa8LrWa4bk1M+5Cp1rQIu0UVQ
  z6fGCA8nHyLoIJBMEgx8KPacia2W3/phdnbfHASD9i+3lEHGv8hJ40itorKM9YhL
  h6VLXmD/bZhgkj8rUGIKogUqx9ufqOK+8e+bEMyB44ZrNhF9mGnys+eJt3a5VGvS
  RwEM2FfXMwKr26AmiumUsqXYA4SvcLk97ytjXdUzdcR1NoHeGEpb2FmM4aOx0zQK
  Hzl0JiGwNaCbsIwndErFg3Hrj/trheUn
  =17r4
  -----END PGP MESSAGE-----
password: |
  -----BEGIN PGP MESSAGE-----
  Version: GnuPG v2.0.22 (GNU/Linux)
  hQIMA+BNIwZHcIj9ARAA3lkQuguW9ZzUAa6kN5K8IPRViCSm3fPOINYi7mbgG6+Z
  gQEqbFOiEQNLXNoDUWaw1847TZ0Lu6hVO9WCq1xlOZkN90u8xG8UCvsTmBcOhQ9g
  /MA4BpdzllbW6I3yXpk1dA5b7F4Em8c3ii6jyTENfJPsehBYEbynxbRiZ8YuohAw
  bKgXMekuWJodFoyaSMupDhAfYDpxK+gNqg3QoF4kx1etbqdOxooXWh31e/W3jZwd
  5n8qDz7uamZCRzK/YvsckrqKzDByRxsziUk5BA8KiYbDaiZuqHzPR1H/q5R4gtCE
  n0v75E/dhXyqiOG7IyQukwW4dTPGpBngtIPUzwCsD/qeNK0Gt6EBzk1+SVbSu2so
  SscJXb+uQZsbELRKuypEF2IVpA8I9xIW23ugeNcXO0sfWixk1040CbKaXRqHARUz
  6i2aMan8bYpS6MEFXCR21rrcgaCxd0fSwkAVTnnCHvTP2PBi/CA0IfNhGe21KBT7
  torb2pinlW7a3vTeNGmIEztOgkxxbP2L0Pl5Ul9Fp4COMTPyYiyRZ5ZkCXrlQhIp
  1TB0OZ2JbfrEx3sZJOZxnFSnvXHAWsvMZxEIdaxXbVyp9mWe+Mo+/QrijGVg0emp
  lTKV9+lva1kwvbXwdYEbA+9giG6JjmXfr2JphhrvwVrieBYLviiZcnirE4l+P7DS
  VgEEYc3HWxFWpBczYql/810gRia9rUEeD0jf8xvChDMbAUn60Wf5LgngyVbtnxut
  HrIkHIhjLhWdV7tVyx9VPV43nK5O9c62wn0VxL+RjAXzFZt9xMR/
  =89eE
  -----END PGP MESSAGE-----

[denz7201]

I will eventually migrate to Thycotic Secret Server for secrets storage but I am still using this method and felt it appropriate to file the bug report. @cingeyedog when did you run across this if you don't mind me asking?

[cingeyedog]

@denz7201 We are currently on an out of support version of saltstack (2016.11) in OL6 (RHEL6-based). We started testing with 2019.2 and Python3 on OL7 and noticed it then (end of November 2019). I haven't tested it, but I assume this bug has existed ever since salt moved to support Python3.

During the upgrade testing process, I found the different SDB modules in the docs and it worked. Since it supports gpg-encrypted blobs, we found it to be an advantage to be able to store the yaml files in git. (We are researching how to build salt-masters quickly and need a place to store these secrets. Dont want to get stuck again in an unsupported salt version).

However, in November, we got busy with other projects and stopped working on the salt upgrade project (and honestly, I forgot to submit an issue report about sqlite3 SDB. Thank you for doing that). Then, with the recent CVEs, we started up the upgrade project again. This time, we HAVE to upgrade the system. When we built the new servers and installed 3000.2 with Python3, we just automatically did it with the yaml based sdb.

[denz7201]

Your story sounds similar to mine. I am rolling our fleet forward to centos 8 from 7. The current salt master is on centos 7 w/ python.2.7. That server is getting long in the tooth having survived a junior co-workers learning-curve. My plan is to move to Centos 8 and then work on creating an sdb module for secret server. I don't want to re-tool twice so I may just pause here til our secret-server turn up is complete. Curious that this hasn't been made more of an issue given the relative ease of using a sqlite store.

[denz7201]

@waynew Did you plan on following up here or have you given things the kiss of death " Info Needed " ?

[sagetherage]

We do really need to do better here, I am looking into it, kicking this back to triage to get follow up, today.

[sagetherage]

@nicholasmhughes if you have any input please do respond, as I attempt follow up, thank you! @saltstack/team-cloud

[denz7201]

We do really need to do better here, I am looking into it, kicking this back to triage to get follow up, today.

Thanks Sage. Last I checked this was still an issue.

[nicholasmhughes]

not really a cloud issue, but something like this might do the trick:

diff --git a/salt/sdb/sqlite3.py b/salt/sdb/sqlite3.py
index cf2095ce75..420f9ceae7 100644
--- a/salt/sdb/sqlite3.py
+++ b/salt/sdb/sqlite3.py
@@ -156,4 +156,10 @@ def get(key, profile=None):
     res = res.fetchone()
     if not res:
         return None
-    return salt.utils.msgpack.unpackb(res[0])
+    ret = salt.utils.msgpack.unpackb(res[0])
+    log.debug("sdb return value is: %s", ret)
+    try:
+        ret = ret.decode("utf-8")
+    except (UnicodeDecodeError, AttributeError):
+        pass
+    return ret

If I have time to track down tests and write one, I'll put in a PR... or if anyone wants to steal this code... go for it. :wink: