Open Vladimir-csp opened 10 months ago
Thanks for reporting this @Vladimir-csp , I have been looking at it, but I've been very short on time. I believe the changes, ultimately, are related to the change in hvac
: https://github.com/hvac/hvac/pull/1021
And since this collection does use a custom session, we probably have to adjust for those changes. I still don't have a 100% complete picture yet, and I can't actually say if this truly only affects FreeBSD (it shouldn't).
@Vladimir-csp can I ask if you are also using the retries
option?
Yes, retries are set via ansible_hashi_vault_retries
Thank you @Vladimir-csp , can you also show a dump of your env vars (non-sensitive values of course).
As I look through the code, the changes we made in hvac
should not be affecting our use case, even when retries are being used (I asked about that because it's the only time we use a custom session). But of course, the change is suspicious, it lines up, I just can't figure out (yet) how it would end up in your situation.
If you try without retries, does that fix the CA issue or is it still present?
I will be able to check in a week+ or so, thanks.
Yes, with ansible_hashi_vault_retries: ~
it works with hvac 2.0
(but still requires explicitly setting ansible_hashi_vault_ca_cert: /etc/ssl/certs
)
thank you @Vladimir-csp
can you also show a dump of your env vars (non-sensitive values of course)
_=/usr/bin/env
ANSIBLE_LOG_PATH=/dev/null
BLOCKSIZE=K
EDITOR=mcedit
ENV=/data/home/username/.shrc
HOME=/data/home/username
LANG=en_GB.UTF-8
LC_ALL=en_GB.UTF-8
LOGNAME=username
MAIL=/var/mail/username
OLDPWD=/data/home/username
PAGER=less
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/data/home/username/bin
PS1=\[\033[01;32m\]\u\[\033[00m\]@\[\033[01;35m\]\h\[\033[00m\]:\[\033[01;34m\]\w\n\[\033[01;00m\]\$
PWD=/data/home/username/git/[...]/ansible
SHELL=/usr/local/bin/bash
SHLVL=1
SSH_CLIENT=[...]
SSH_CONNECTION=[...]
SSH_TTY=/dev/pts/0
TERM=xterm-256color
USER=username
there was also VAULT_ADDR=https://...
, but it does not make any difference.
Are there any news or estimates on this?
Unfortunately not, I started looking at it back then, but I have not been able to find the time since then due to other priorities in the collection and mostly lack of time in general.
I still think this is worthwhile to work on but I can't offer any estimate. If you or anyone would like to try to investigate further or put up a fix I would certainly encourage that too.
I wanted to add my experience to this issue. I'm seeing the same behavior when upgrading from hvac 1.2.1
to hvac 2.1.0
or hvac 2.2.0
in an Ubuntu 20.04/22.04
WSL image. We have a Vault setup secured using an internal certificate authority. As part of our Ansible Vault lookups, we passed the internal CA cert using the cacert
/ca_cert
parameter. This is no longer being respected, and calls to Vault are failing with this message:
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
unable to get local issuer certificate (_ssl.c:1006)'))).
HTTPSConnectionPool(host='vaultserver.domain.com', port=443):
Max retries exceeded with url: /v1/engine/data/path/to/credential
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')
The stack trace shows failure with requests
, and requests
uses certifi
package. I'm able to get things working again by adding our internal CA to the certifi
cacert.pem
located in the certifi
package: .local/lib/python3.11/site-packages/certifi/cacert.pem
Certifi
doesn't support addition/removal of certificates and this isn't a valid long-term workaround (certifi updates will break this again).
So for now, the only workaround is to keep hvac
at version 1.2.1
until this is addressed in community.hashi_vault
.
As @briantist pointed out, this issue stems from hvac changes to session
And here's the hvac outline of breaking changes planned for 2.x and 3.x
hvac
guidance for making use of a private CA
I think the changes to community.hashi_vault need to be made here, but I'm not familiar enough with python to say for sure (or offer a fix). :(
@daniel-chamberland thanks very much for the additional information, I'm still very short on time until at least later this month, but I do hope to track down the cause and fix, whether it's in this collection or in hvac
or both.
I am running into the same issue as above with hvac 2.1.0
on Ubuntu 22.04.4 LTS
using ansible [core 2.15.10]
.
As a workaround I'll use ansible.builtin.uri
to make the API calls.
If any additional information is needed please let me know.
SUMMARY
After upgrading
py39-hvac
from 1.2.1 to 2.0.0 vault modules fail withrequests.exceptions.SSLError
due toCERTIFICATE_VERIFY_FAILED
ISSUE TYPE
COMPONENT NAME
hvac
ANSIBLE VERSION
COLLECTION VERSION
CONFIGURATION
OS / ENVIRONMENT
FreeBSD 13
STEPS TO REPRODUCE
Set variables in context of vault module or lookup:
Try accessing vault via any vault module.
EXPECTED RESULTS
It works. Preferably it should work even without
ansible_hashi_vault_ca_cert
, since/etc/ssl/certs
is a system trust store.ACTUAL RESULTS
Underlying python
requests
module fails to verify HTTPS server, despite CA exists in system's trust store (hvac 1.2.1), and despiteansible_hashi_vault_ca_cert
points it to system trust store explicitly (hvac 2.0.0).