tiandrey
commented
3 years ago Update: with puppetserver version 6.2.0-1stretch, there were only Vault::ConnectionPool::PoolShuttingDownError errors (about 100 errors per day)
However, with puppetserver version 6.15.1-1stretch (I tested both old single-threaded and new multithreaded JRuby mode), there are many different errors, here are stats for one day:
root@puppet:~# zcat /var/log/puppetlabs/puppetserver/puppetserver-2021-03-25.0.log.gz /var/log/puppetlabs/puppetserver/puppetserver-2021-03-25.1.log.gz | grep 'Skipping backend' | grep -o '\[hiera-vault\].*' | awk -F ' on node' '{print $1}' | awk -F ' \\(file:' '{print $1}' | sort | uniq -c | sort -n
3 [hiera-vault] Skipping backend. Configuration error: Socket closed
3 [hiera-vault] Skipping backend. Configuration error: undefined method `request' for nil:NilClass
3 [hiera-vault] Skipping backend. Configuration error: undefined method `ssl_version=' for nil:NilClass
12 [hiera-vault] Skipping backend. Configuration error: The Vault server at `https://127.0.0.1:8200' is not currently
81 [hiera-vault] Skipping backend. Configuration error: Vault::ConnectionPool::PoolShuttingDownError
2700 [hiera-vault] Skipping backend. Configuration error: undefined method `configure' for nil:NilClass
3087 [hiera-vault] Skipping backend. Configuration error: no cipher matchSome of no cipher match errors may be accounted for changes in default ciphers in puppetserver, but surely not all of them.
Also, https://127.0.0.1:8200 is NOT address of my Vault server.
petems
hfrog
I have a heavy-loaded puppetserver (about 2k nodes with default 30m checkin interval; 32 workers) with lots of environments (20-30). Relatively frequently I receive different errors caused by
$vaultbeing shut down or nil'ed. I guess this one occurs when$vault.shutdownhappens at the same time as lookup is made for another node:And this one, I guess, happens when
$vault = nilduring another lookup:To be honest, for this one I'm not sure what causes it, but maybe it has something to do with simultaneous shutdown:
For now I solved this problem by removing global vault caching object and changing it back to local variable; adding
vault.shutdowninstead of$shutdown.callseems to solve problem of CLOSE_WAIT sockets (although right now I see one such socket on my server). Is there any better way to do this? Maybe fence these operations with mutex or something similar?