Adding ACLS / Policies failes with unable to get local issuer certificate -> Puppet 6 / LetsEncrypt

Affected Puppet, Ruby, OS and module versions/distributions

Puppet: 6.28.0
Distribution: Debian Bullseye
Module version: 7.3.1

How to reproduce (e.g Puppet code you use)

  # Loop for create the default ACLs
  $consul_acl_policies = ['acl-replication','agent','query','service','session','node']

  # First for READ
  $consul_acl_policies.each | String $consul_acl_policy| {

    consul_policy { "${consul_acl_policy}-read":
      description   => " ${consul_acl_policy}-read, generated by puppet",
      rules         => [ { 'resource'    => "${consul_acl_policy}_prefix", 'segment'     => '', 'disposition' => 'read' }, ],
      acl_api_token => $acl_api_token,
      protocol      => https,
      port          => 8501,
      hostname      => $::fqdn,
    }
  }

What are you seeing

Notice: /Stage[main]/Consul::Install/File[/opt/consul/archives]/ensure: created (corrective)
Notice: /Stage[main]/Consul::Install/File[/opt/consul/archives/consul-1.14.4]/ensure: created (corrective)
Notice: /Stage[main]/Consul::Install/Archive[/opt/consul/archives/consul-1.14.4.zip]/ensure: download archive from https://releases.hashicorp.com/consul/1.14.4/consul_1.14.4_linux_amd64.zip to /opt/consul/archives/consul-1.14.4.zip and extracted in /opt/consul/archives/consul-1.14.4 with cleanup (corrective)
Notice: /Stage[main]/Consul::Install/File[/opt/consul/archives/consul-1.14.4/consul]/mode: mode changed '0755' to '0555' (corrective)
Notice: /Stage[main]/Consul::Run_service/Service[consul]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Consul::Run_service/Service[consul]: Unscheduling refresh on Service[consul]
Warning: Cannot retrieve ACL token list: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
Notice: /Stage[main]/Profile::Consul::Cluster/Consul_policy[acl-replication-read]/ensure: created (corrective)
Warning: Unable to create policy acl-replication-read: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
Error: /Stage[main]/Profile::Consul::Cluster/Consul_policy[acl-replication-read]: Could not evaluate: Got nil value for id
Notice: /Stage[main]/Profile::Consul::Cluster/Consul_policy[agent-read]/ensure: created (corrective)

extra

...
    class { 'consul':
      version          => $version,
      pretty_config    => true,
      config_dir       => '/etc/consul',
      service_enable   => $service_enable,
      service_ensure   => $service_ensure,
      extra_groups     => ['ssl-cert'],
      acl_api_hostname => $::fqdn,
      acl_api_protocol => https,
      acl_api_port     => 8501,
      config_hash      => {
        'bootstrap_expect'            => 3,
        'data_dir'                    => '/opt/consul',
      'datacenter'                  => "$datacenter",
        'node_name'                   => $facts['networking']['hostname'],
        'server'                      => true,
        'domain'                      => 'consul.',
        'ui_config'                   => { 'enabled' => $enable_ui, },
        'log_level'                   => $log_level,
        'enable_syslog'               => $enable_syslog,
        'leave_on_terminate'          => false,
        'skip_leave_on_interrupt'     => false,
      'check_update_interval'       => "5m",
        'disable_update_check'        => true,
        #'client_addr'                => $facts['networking']['ip'],
        'client_addr'                 => $listen_address,
        'advertise_addr'              => $listen_address,
        'cert_file'                   => $tls_cert_file,
        'key_file'                    => $tls_key_file,
        'ca_path'                     => '/usr/share/ca-certificates/',
        'tls_min_version'             => 'tls12',
      'tls_cipher_suites'           => 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384',
        'disable_anonymous_signature' => false,
...

I have no idea .. what is missing ..

root@linode-ffm-hashi-server-01:[~]: curl https://$(hostname -f):8501
<a href="/ui/">Moved Permanently</a>.

It works on other places .. but the only diff is .. that I have here two network interfaces with different IPs. (public / private) ... I have no idea anymore .. where to search.

any suggestions ?

cu denny

hi,

I tried to add debug .. and added to ./lib/puppet_x/consul/acl_base.rb

  module PuppetX::Consul::ACLBase
    class BaseClient
      def initialize(hostname, port, protocol, api_token = nil)
        @global_uri = URI("#{protocol}://#{hostname}:#{port}/v1/acl")
        @http_client = Net::HTTP.new(@global_uri.host, @global_uri.port)
        @http_client.use_ssl = true if @global_uri.instance_of? URI::HTTPS
        @api_token = api_token
        puts @global_uri
      end

...

to see, what is called .. and the printed URL is correct .. than I tried to use the Ruby from Puppet ..

require "net/https"
require "uri"

uri = URI.parse("https://linode-ffm-hashi-server-01.sandbox.example.com:8501/v1/acl")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

request = Net::HTTP::Get.new(uri.request_uri)

response = http.request(request)
puts response.body

and the output:

Traceback (most recent call last):
    6: from /root/test.ry:11:in `<main>'
    5: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:1458:in `request'
    4: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:909:in `start'
    3: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
    2: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/http.rb:985:in `connect'
    1: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `ssl_socket_connect'
/opt/puppetlabs/puppet/lib/ruby/2.5.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError)

which is exactly the same .. I get with Puppet :-)

Consul config has:

    "ca_file": "/etc/ssl/acme/fullchain.pem",
    "cert_file": "/etc/ssl/acme/fullchain.pem",
     "key_file": "/etc/ssl/acme/privkey.pem"

after playing a lot .. and searching .. I've found the root cause: Puppet 6 on Debian brings

/opt/puppetlabs/puppet/bin/ruby -v
ruby 2.5.9p229 (2021-04-05 revision 67939) [x86_64-linux]

So, I've upgraded to Puppet7

root@linode-ffm-hashi-server-01:[/etc/apt/sources.list.d]: /opt/puppetlabs/puppet/bin/ruby -v
ruby 2.7.7p221 (2022-11-24 revision 168ec2b1e5) [x86_64-linux]

and ...

root@linode-ffm-hashi-server-01:[/etc/apt/sources.list.d]:  /opt/puppetlabs/puppet/bin/ruby /root/test.ry 
Invalid URL path: not a recognized HTTP API endpoint

Success !

Now, also Puppet runs fine and does not drop anymore issues.

voxpupuli / puppet-consul