multiple problems: params.dh, private.key, duplicate fullchain_with_key.pem.

[advorkin]

Hello,

puppet6.example.com - puppet (ACME) master reflect.example.com - puppet client

After our initial installation, we got the following error message on the master:

puppet6.example.com puppet-agent[13881]: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'File[/etc/acme.sh/configs/reflect.example.com/params.dh]' in parameter 'require' (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/deploy/crt.pp, line: 97) on node puppet6.example.com

puppet6.example.com puppet-agent[13881]: (/Stage[main]/Acme/Acme::Request::Crt[reflect.example.com]/Acme::Deploy::Crt[reflect.example.com]/Concat[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]/Concat_file[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]) Failed to generate additional resources using 'eval_generate': Could not retrieve source(s) /etc/acme.sh/keys/reflect.example.com/private.key

params.dh as far as I understand should only be on the node itself (not on the master). The same goes for private.key. With the way source code is written right now, puppet is expecting both files to on both - master and node and is looking for /etc/acme.sh/keys/reflect.example.com/private.key, which of course doesn't exist on the master.

I added the following lines in deploy/crt.pp, which fixed that problem.

if $::hostname == 'puppet6' {
      $dh = "${cfg_dir}/puppet6.example.com/params.dh"
      $key = "${key_dir}/puppet6.example.com/private.key"
      $crt_full_chain_with_key = "${key_dir}/puppet.example.com/fullchain_with_key.pem"
}
else {
      $dh = "${cfg_dir}/${domain}/params.dh"
      $key = "${key_dir}/${domain}/private.key"
      $crt_full_chain_with_key = "${key_dir}/${domain}/fullchain_with_key.pem"
}

At this point I am experiencing the following issue:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Concat[/etc/acme.sh/keys/puppet.example.com/fullchain_with_key.pem] is already declared at (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/deploy/crt.pp, line: 78); cannot redeclare (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/deploy/crt.pp, line: 78) (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/deploy/crt.pp, line: 78, column: 3) (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/request/crt.pp, line: 32) on node puppet6.example.com

I have tried a couple of things, but none of them work 100% as often times fullchain_with_key.pem doesn't get distributed at all.

I would really appreciate your help on this matter.

Thank you!

[fraenki]

puppet6.example.com - puppet (ACME) master reflect.example.com - puppet client

Please provide the Puppet manifests (or hiera config) that you are using on your puppet master and puppet client.

[advorkin]

Thank you.

$ cat reflect.pp

# defines profile to request acme certificate
class profiles::acme::reflect {
    include ::acme
    $fqdn = downcase($::fqdn);

    ::acme::certificate { $::fqdn:
            use_profile    => 'nsupdate',
            dh_param_size  => 2048,
            acme_host      => 'puppet6.example.com',
            use_account    => 'myemail@email.com',
            letsencrypt_ca => 'staging';
    }

$ cat master.pp

#defines profile for acme master
class profiles::acme::master {

    class { 'acme':
      accounts => ['myemail@email.com'],
      profiles => {
        nsupdate => {
          challengetype => 'dns-01',
          hook          => 'nsupdate',
          env           => {
            'NSUPDATE_SERVER' => 'dns1.example.com'
          },
          options       => {
            dnssleep        => 5,
            nsupdate_id     => 'acme-update',
            nsupdate_type   => 'hmac-sha512',
            challenge_alias => '_acme.example.com',
            nsupdate_key    => 'mykey',
          }
        }
      }
    }
}

[advorkin]

There is a chance I created a duplicate error message myself with my configuration, but I just started fresh and still getting error message after my second run on master:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'File[/etc/acme.sh/configs/reflect.example.com/params.dh]' in parameter 'require' (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/deploy/crt.pp, line: 104) on node puppet6.example.com

[advorkin]

I was able to comment out this require, which fixed above problem.

Now next one: Error:

Why puppet master is looking for fullchain_with_key.pem from a client?

/Stage[main]/Acme/Acme::Request::Crt[reflect.example.com]/Acme::Deploy::Crt[reflect.example.com]/Concat[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]/File[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory @ rb_sysopen - /etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem

[fraenki]

I was able to comment out this require, which fixed above problem.

I'm not going to analyze this further with modified sources. I'll look into the root problem and try to reproduce it first.

[advorkin]

Ok thank you.

I think... it's resolved by specifying a hostname as opposed to $::fqdn

::acme::certificate { $::fqdn:

changed to

::acme::certificate { 'reflect.example.com':

Please let me know what you think and thank you for your help.

[advorkin]

I take my words back. It's still not working. Thank you.

[advorkin]

Error: /Stage[main]/Acme/Acme::Request::Crt[reflect.example.com]/Acme::Deploy::Crt[reflect.example.com]/Concat[/etc/acme.sh/certs/reflect.example.com/fullchain.pem]/Concat_file[/etc/acme.sh/certs/reflect.example.com/fullchain.pem]: Failed to generate additional resources using 'eval_generate': Could not retrieve source(s) /etc/acme.sh/configs/reflect.example.com/params.dh Error: /Stage[main]/Acme/Acme::Request::Crt[reflect.example.com]/Acme::Deploy::Crt[reflect.example.com]/Concat[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]/Concat_file[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]: Failed to generate additional resources using 'eval_generate': Could not retrieve source(s) /etc/acme.sh/keys/reflect.example.com/private.key Error: Could not set 'file' on ensure: No such file or directory @ rb_sysopen - /etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem Error: Could not set 'file' on ensure: No such file or directory @ rb_sysopen - /etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem Wrapped exception: No such file or directory @ rb_sysopen - /etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem Error: /Stage[main]/Acme/Acme::Request::Crt[reflect.example.com]/Acme::Deploy::Crt[reflect.example.com]/Concat[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]/File[/etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory @ rb_sysopen - /etc/acme.sh/keys/reflect.example.com/fullchain_with_key.pem

[fraenki]

@advorkin I was unable to reproduce this issue. Are you sure that you are using the latest version of this module? A duplicate declaration error was fixed back in december (#20).

[fraenki]

For reference, this is the configuration I was using in a test environment.

Puppetserver 6.7.1 Puppet Agent 6.10.1

Config on Puppetserver:

acme::manage_packages: false
acme::letsencrypt_ca: 'staging'
acme::accounts:
  - 'acme@example.com'
acme::profiles:
  'variomedia':
    challengetype: 'dns-01'
    hook: 'variomedia'
    env:
      VARIOMEDIA_API_TOKEN: 'secret'
    options:
      dnssleep: 120

Config on client:

acme::manage_packages: false
acme::certificates:
  '%{::fqdn}':
    use_profile: 'variomedia'
    use_account: 'acme@example.com'
    letsencrypt_ca: 'staging'

No errors with this configuration and puppet-acme version 1.0.4.

[advorkin]

Yes, I am definitely using the latest version. Duplicate issue was due to my "fix", which was wrong.

My issue is that master (puppet6) looks for private.key and fullchain_with_key.pem on master for an agent (reflect), which shouldn't be the case. Should keys be stored in /certs on master for an agent as well?

[fraenki]

My issue is that master (puppet6) looks for private.key and fullchain_with_key.pem on master for an agent (reflect), which shouldn't be the case.

True. And I have no idea how this could happen. Like I said, I cannot reproduce it. Verify that you run only the required code on your master and that PupeptDB is properly setup.

Should keys be stored in /certs on master for an agent as well?

No, as pointed out in the README:

Keep your private keys safe on the host they belong to and let your Puppet Server sign the CSRs and distribute the certificates.

[advorkin]

Thank you. But certificates are being copied there by the module. I don't copy them myself.

[fraenki]

But certificates are being copied there by the module. I don't copy them myself.

Well, what? o.O

[advorkin]

I don't copy certs into /certs directory or any other directories on the master. Everything is done by the master. If you'd like, I can include the logs.

[fraenki]

Ah, you are talking about keys, not certs.

If you'd like, I can include the logs.

Sure. Cleanup all old traces first and start over. Then please provide the full output of a Puppet Run on your master. Ensure that you don't run your profiles::acme::reflect class on the master.

[advorkin]

root@puppet6 /tmp $ ls -la /etc/acme.sh/certs/ total 16 drwxr-xr-x. 4 acme acme 4096 Mar 30 09:50 . drwxr-xr-x. 9 acme acme 4096 Jan 27 08:40 .. drwxr-xr-x. 2 acme acme 4096 Mar 30 09:12 puppet6.example.com drwxr-xr-x. 2 acme acme 4096 Mar 30 09:50 reflect.example.com root@puppet6 /tmp $ ls -la /etc/acme.sh/certs/* /etc/acme.sh/certs/puppet6.example.com: total 36 drwxr-xr-x. 2 acme acme 4096 Mar 30 09:12 . drwxr-xr-x. 4 acme acme 4096 Mar 30 09:50 .. -rw-r--r--. 1 root acme 1029 Jan 27 08:40 cert.csr -rw-r--r--. 1 root acme 488 Mar 27 13:03 cert.ocsp -rw-r--r--. 1 root acme 1919 Mar 27 13:03 cert.pem -rw-r--r--. 1 root acme 1678 Jan 27 11:29 chain.pem -rw-r--r--. 1 root acme 4021 Mar 30 09:12 fullchain.pem -rw-r-----. 1 root acme 5696 Mar 23 16:47 fullchain_with_key.pem

/etc/acme.sh/certs/reflect.example.com: total 20 drwxr-xr-x. 2 acme acme 4096 Mar 30 09:50 . drwxr-xr-x. 4 acme acme 4096 Mar 30 09:50 .. -rw-r--r--. 1 acme acme 1919 Mar 30 09:50 cert.pem -rw-r--r--. 1 acme acme 1680 Mar 30 09:50 chain.pem -rw-r--r--. 1 acme acme 3599 Mar 30 09:50 fullchain.pem

puppetacme_1strun.txt

puppetacme_2ndrun.txt

[advorkin]

Thank you very much for your help. Please let me know if you need any additional info.

[advorkin]

My puppet master role includes two profiles: master and client (NOT reflect). I am attaching client below. It's using $::fqdn, NOT reflect.example.com.

class profiles::acme::client {

include ::acme

$fqdn = downcase($::fqdn);

::acme::certificate { $::fqdn:
        use_profile    => 'nsupdate',
        dh_param_size  => 2048,
        acme_host      => 'puppet6.example.com',
        use_account    => 'email@example.com',
        letsencrypt_ca => 'staging';
}

[advorkin]

From acme.log

[Mon Mar 30 10:50:25 EDT 2020] Installing cert to:/etc/acme.sh/certs/reflect.example.com/cert.pem [Mon Mar 30 10:50:25 EDT 2020] Installing CA to:/etc/acme.sh/certs/reflect.example.com/chain.pem [Mon Mar 30 10:50:25 EDT 2020] Installing full chain to:/etc/acme.sh/certs/reflect.example.com/fullchain.pem

[advorkin]

I had to reboot the server (master) and after it was done, no more problems it seems... Please ignore my ticket in the meantime as I am not sure what happened. I'm sorry for all the trouble. I will close it for now. Stay safe and wish you all the best.

[fraenki]

@advorkin Please test https://github.com/fraenki/puppet-acme/commit/312f4b05301da75a2e3ea6cc876b26d378739006, it should fix your issue. The commit message hints at the root cause for this issue, which hasn't been discovered before.

[advorkin]

Thank you! I will test it out and let you know.

[advorkin]

So with my testing:

If puppet master roles includes only master profiles, reflect's certs and keys get created and properly distributed. Puppet master does not get its own certs and keys.

root@puppet6 /etc/acme.sh/results $ ls reflect.example.com reflect.example.com reflect.example.com

If I add the client's profile to puppet master roles (in order to create certs and keys on master):

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'File[/etc/acme.sh/configs/reflect.example.com/params.dh]' in parameter 'require' (file: /etc/puppetlabs/code/environments/puppet6/site-modules/acme/manifests/deploy/crt.pp, line: 97) on node puppet6.example.com

[advorkin]

So the issue starts when puppet master wants to be a client as well. If it only serves OTHER clients, $::acme_crts gets set properly and everything works as expected in both versions of your code. Your new code did not make a difference, I am sorry.

Debug: Facter: fact "acme_crts" has resolved to "reflect.example.com".

The way you distribute keys i based on this fact:

require 'facter' crt_domains = Dir['/etc/acme.sh/results/*.pem'].map { |a| File.basename(a, '.pem') }

Facter.add(:acme_crts) do setcode do

With puppet6.example.com files appearing in results directory, it causes for the fact to look for puppet6.example.com {domain} and deploy private.key, params.dh; concat fullchain_with_key.pem, which, obviously, do not exist on the master (puppet6) for the client (reflect). Hope it makes sense. If a zoom session will help you with troubleshooting, please let me know. Thank you for all your help.

[fraenki]

So the issue starts when puppet master wants to be a client as well. If it only serves OTHER clients, $::acme_crts gets set properly and everything works as expected in both versions of your code.

The new code does exactly solve this issue. I was able to reproduce your issue and verify the fix multiple times.

I assume you're still not using the new code, maybe the old code is still cached by your Puppetserver.

[advorkin]

Okay, I will try again later tonight to make sure and will double check everything.

Should puppet master be getting a client profile?

[advorkin]

I've tried again with the same result. This is when I add client profile for master (puppet6) after installing client (reflect).

As soon as I add client profile, I get an error message.

Again, this is what my client profile looks like:

$ cat profiles/manifests/acme/client.pp
# defines profile to request acme certificate
#defines the puppetmaster6 role

class profiles::acme::client {

    include ::acme

    $fqdn = downcase($::fqdn);

    ::acme::certificate { $::fqdn:
            use_profile    => 'nsupdate',
            dh_param_size  => 2048,
            acme_host      => 'puppet6.example.com',
            use_account    => 'email@email.com',
            letsencrypt_ca => 'staging';
    }

I've made sure to clear the cache and checked on the master under correct environment.

My role for puppet master. Could I be missing something that it fixed your installation?

$ cat roles/manifests/puppetmaster6.pp
#defines the puppetmaster6 role
class roles::puppetmaster6 {
    .......
    include profiles::acme::master
    include profiles::acme::client
}

[advorkin]

Temporary solution is to actually first add puppet6 with client profile (so to get all keys and certs). Then to remove the client profile from puppet master roles and add reflect.example.com and whatever other clients are necessary. Hope I am making sense :)

[fraenki]

Until now all examples were incomplete and I had to figure out how to reproduce your issue myself. That was rather time consuming for me.

I have to ask you to provide a complete, self-contained example configuration that allows me to reliably reproduce this issue. I will not use your profile code to reproduce this issue. The examples should be pure Puppet manifests (and/or Hiera data), similar to the examples provided in the README.

[advorkin]

$ cat roles/manifests/puppetmaster6.pp
#defines the puppetmaster6 role
class roles::puppetmaster6 {
   .......
    include profiles::acme::master
    include profiles::acme::client
}

$ cat profiles/manifests/acme/master.pp
#defines profile for acme master

class profiles::acme::master {

    class { 'acme':
      accounts => ['email@email.com'],
      profiles => {
        nsupdate => {
          challengetype => 'dns-01',
          hook          => 'nsupdate',
          env           => {
            'NSUPDATE_SERVER' => 'dns1.example.com'
          },
          options       => {
            dnssleep        => 5,
            nsupdate_id     => 'acme-update',
            nsupdate_type   => 'hmac-sha512',
            challenge_alias => '_acme.example.com',
            nsupdate_key    => 'secret',
          }
        }
      }
    }
}

$ cat profiles/manifests/acme/client.pp
# defines profile to request acme certificate

class profiles::acme::client {

    include ::acme

    $fqdn = downcase($::fqdn);

    ::acme::certificate { $::fqdn:
            use_profile    => 'nsupdate',
            dh_param_size  => 2048,
            acme_host      => 'puppet6.example.com',
            use_account    => 'email@email.com',
            letsencrypt_ca => 'staging';
    }

    $base_dir = $::acme::params::base_dir
    $crt_dir  = $::acme::params::crt_dir
    $key_dir  = $::acme::params::key_dir

    file {
        "/etc/ssl/${fqdn}.crt":
            ensure    => file,
            owner     => 'root',
            group     => 'root',
            mode      => '0644',
            source    => "${::acme::params::crt_dir}/${fqdn}/cert.pem",
            subscribe => Acme::Certificate[$fqdn];
        "/etc/ssl/${fqdn}.key":
            ensure    => file,
            owner     => 'root',
            group     => 'root',
            mode      => '0640',
            source    => "${::acme::params::key_dir}/${fqdn}/private.key",
            subscribe => Acme::Certificate[$fqdn];
        "/etc/ssl/${fqdn}.ca":
            ensure    => file,
            owner     => 'root',
            group     => 'root',
            mode      => '0644',
            source    => "${::acme::params::crt_dir}/${fqdn}/chain.pem",
            subscribe => Acme::Certificate[$fqdn];
    }

}

$ cat roles/manifests/reflect.pp
#defines the role of a reflect server
class roles::reflect {
    .....
    include profiles::acme::client
}

hosts.pp

node 'puppet6.example.com'
{
    include roles::puppetmaster6
}

[fraenki]

I have to ask you to provide a complete, self-contained example configuration that allows me to reliably reproduce this issue. I will not use your profile code to reproduce this issue.

[advorkin]

This is what I did. I am not sure what else you are asking for. Are you expecting a tar file?

[fraenki]

I still cannot reproduce this so I'll close this issue for now. If someone is able to explain what went wrong in these examples or is able to make them reproducable to me, I'll gladly reopen this issue.

[philipfreude]

This sounds a lot like #31. I saw a the same behavior with the params.dh on my puppetserver.

Are you open to try https://github.com/phifre/puppet-acme (master) and report if this fixes the issue with your original configuration?

[fraenki]

The root cause was identified in #31 and a fix was released in version 2.2.0.