Puppet with Mutliple Compile Masters

[frankmundt]

Puppet allows for multiple compile masters however the Puppet Forge Conjur Module does not support that feature. The Conjur module is using the Puppet Host private/public key pair to encrypt/decrypt the Puppet Fact conjur.encrypted.token. Since the pair are different for each Puppet Compile master, the Conjur module fails as soon as it runs on a different compile master.

The facter/conjur.rb and decrypt.rb function need to be modified to use something other than the host public/private key pair for encryption/decryption.

[sgnn7]

Hey @frankmundt, Thank you for reporting this issue! :)

Do you mind providing us with a bit more info on:

• [ ] What version of the module are you running?
• [ ] What version of Puppet master are you running?
• [ ] What kind of infrastructure topology are running to recreate this error?
• [ ] How frequently does the problem occur in this test environment?
• [ ] If it is infrequent, is it possible to make it occur more reliably?

Apologies if these sound like somewhat boilerplate questions but they should help us make sure that we're able to quickly reproduce your target env if possible. I think I understand the issue but I'm guessing that you're not running a very common topology so I'll wait for you to answer back on that and then we can add it to the work queue.

Thanks, Srdjan

[frankmundt]

The Puppet forge module is CyberArk/Conjur 2.0.2, I will be testing with 2.0.4 next week. We are running Puppet Enterprise 2019.1 (Puppet Version 6.14.0). Our topology is a Puppet Master and an HA replica ( in standby mode ) behind a load balancer. We are attempting to add 2 Puppet Compile Masters behind a load balancer and that is when we discovered this issue. We stopped the deployment of the Compile Masters. As soon as I point a Puppet Agent to the compile master the catalog compile fails in the decrypt.rb function because it cannot decrypt the encrypted_token Puppet Fact. It happens with every Puppet Agent run when pointing to the compile master load balancer. Switching back to the Puppet Master load balancer corrects the issue.

Thanks, Frank

On Fri, Jul 24, 2020 at 4:07 PM Srdjan Grubor notifications@github.com wrote:

Hey @frankmundt https://github.com/frankmundt, Thank you for reporting this issue! :)

Do you mind providing us with a bit more info on:

• What version of the module are you running?
• What version of Puppet master are you running?
• What kind of infrastructure topology are running to recreate this error?
• How frequently does the problem occur in this test environment?
• If it is infrequent, is it possible to make it occur more reliably?

Apologies if these sound like somewhat boilerplate questions but they should help us make sure that we're able to quickly reproduce your target env if possible. I think I understand the issue but I'm guessing that you're not running a very common topology so I'll wait for you to answer back on that and then we can add it to the work queue.

Thanks, Srdjan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-663734412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN755U2FB4EAYQGUUJ6PTR5HZYHANCNFSM4PHAXPHA .

[sgnn7]

Hey @frankmundt, Apologies for a bit of a delayed response but I had to talk to a couple of people to get a clearer understanding of all the aspects of this issue. Given your latest comments (and interestingly enough) our preliminary Puppet 6 support went in in v2.0.4 so at the very least you should attempt to use that version (or later) for your tests so I'm a bit of at a loss as to how you even got this far with Puppet 6 since it probably didn't have these two fixes as far as I know.

With that said, your issue may still be completely valid as multiple compilation masters are a rather unusual topology for our test cases. I would love it if you can reply back with results from using that new version so that we can be sure it's an issue that still needs fixing. Since we are currently working on support for deferred lookups there is a high likelihood that it will obsolete the need for this certificate encryption completely in v3+ of this module.

PS: If the issue is confirmed with v2.0.4+ and if you are one of our corporate clients, it would also be highly advisable to create a Salesforce enhancement request for this as well so that you can asynchronously track our progress on this and bump up the priority of this work for us too.

Thanks, Srdjan

[frankmundt]

I brought the conjur module inhouse and modified to work with Puppet 6. I just merged in the changes for 2.0.4 and that works fine pointing to the Puppet Master. After modifying the puppet.conf file to use the compile masters I received the following errors:

Conjur Fact failed connecting to https:// with wrong number of arguments (given 3, expected 2).

Error while evaluating a Method call, 'conjur::token' parameter key expects a Sensitive[String] value, got Undef.

We are a corporate client and already have a meeting scheduled to discuss this with our CyberArk team.

On Tue, Jul 28, 2020 at 10:09 AM Srdjan Grubor notifications@github.com wrote:

Hey @frankmundt https://github.com/frankmundt, Apologies for a bit of a delayed response but I had to talk to a couple of people to get a clearer understanding of all the aspects of this issue. Given your latest comments (and interestingly enough) our preliminary Puppet 6 support went in in v2.0.4 so at the very least you should attempt to use that version (or later) for your tests so I'm a bit of at a loss as to how you even got this far with Puppet 6 since it probably didn't have these https://github.com/cyberark/conjur-puppet/pull/92 two https://github.com/cyberark/conjur-puppet/pull/90 fixes as far as I know.

With that said, your issue may still be completely valid as multiple compilation masters are a rather unusual topology for our test cases. I would love it if you can reply back with results from using that new version so that we can be sure it's an issue that still needs fixing. Since we are currently working on support for deferred lookups https://puppet.com/docs/puppet/6.17/deferring_functions.html there is a high likelihood that it will obsolete the need for this certificate encryption completely in v3+ of this module.

PS: If the issue is confirmed with v2.0.4+ and if you are one of our corporate clients, it would also be highly advisable to create a Salesforce enhancement request for this as well so that you can asynchronously track our progress on this and bump up the priority of this work for us too.

Thanks, Srdjan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-665097886, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN7547MXBTSPCEAG47CLDR53S37ANCNFSM4PHAXPHA .

[frankmundt]

I have to add to the last email - the error is happening when connecting to the Puppet master as well.

On Tue, Jul 28, 2020 at 2:23 PM Frank Mundt frankmundt@gmail.com wrote:

I brought the conjur module inhouse and modified to work with Puppet 6. I just merged in the changes for 2.0.4 and that works fine pointing to the Puppet Master. After modifying the puppet.conf file to use the compile masters I received the following errors:

Conjur Fact failed connecting to https:// with wrong number of arguments (given 3, expected 2).

Error while evaluating a Method call, 'conjur::token' parameter key expects a Sensitive[String] value, got Undef.

We are a corporate client and already have a meeting scheduled to discuss this with our CyberArk team.

On Tue, Jul 28, 2020 at 10:09 AM Srdjan Grubor notifications@github.com wrote:

Hey @frankmundt https://github.com/frankmundt, Apologies for a bit of a delayed response but I had to talk to a couple of people to get a clearer understanding of all the aspects of this issue. Given your latest comments (and interestingly enough) our preliminary Puppet 6 support went in in v2.0.4 so at the very least you should attempt to use that version (or later) for your tests so I'm a bit of at a loss as to how you even got this far with Puppet 6 since it probably didn't have these https://github.com/cyberark/conjur-puppet/pull/92 two https://github.com/cyberark/conjur-puppet/pull/90 fixes as far as I know.

With that said, your issue may still be completely valid as multiple compilation masters are a rather unusual topology for our test cases. I would love it if you can reply back with results from using that new version so that we can be sure it's an issue that still needs fixing. Since we are currently working on support for deferred lookups https://puppet.com/docs/puppet/6.17/deferring_functions.html there is a high likelihood that it will obsolete the need for this certificate encryption completely in v3+ of this module.

PS: If the issue is confirmed with v2.0.4+ and if you are one of our corporate clients, it would also be highly advisable to create a Salesforce enhancement request for this as well so that you can asynchronously track our progress on this and bump up the priority of this work for us too.

Thanks, Srdjan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-665097886, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN7547MXBTSPCEAG47CLDR53S37ANCNFSM4PHAXPHA .

[frankmundt]

Here is an update. I have updated to the 2.0.5 version just released and I refreshed the conjur identity and that appeared to resolve the issue on a linux machine. I am going to do more testing and I will update you as soon as I can.

On Tue, Jul 28, 2020 at 4:01 PM Frank Mundt frankmundt@gmail.com wrote:

I have to add to the last email - the error is happening when connecting to the Puppet master as well.

On Tue, Jul 28, 2020 at 2:23 PM Frank Mundt frankmundt@gmail.com wrote:

I brought the conjur module inhouse and modified to work with Puppet 6. I just merged in the changes for 2.0.4 and that works fine pointing to the Puppet Master. After modifying the puppet.conf file to use the compile masters I received the following errors:

Conjur Fact failed connecting to https:// with wrong number of arguments (given 3, expected 2).

Error while evaluating a Method call, 'conjur::token' parameter key expects a Sensitive[String] value, got Undef.

We are a corporate client and already have a meeting scheduled to discuss this with our CyberArk team.

On Tue, Jul 28, 2020 at 10:09 AM Srdjan Grubor notifications@github.com wrote:

Hey @frankmundt https://github.com/frankmundt, Apologies for a bit of a delayed response but I had to talk to a couple of people to get a clearer understanding of all the aspects of this issue. Given your latest comments (and interestingly enough) our preliminary Puppet 6 support went in in v2.0.4 so at the very least you should attempt to use that version (or later) for your tests so I'm a bit of at a loss as to how you even got this far with Puppet 6 since it probably didn't have these https://github.com/cyberark/conjur-puppet/pull/92 two https://github.com/cyberark/conjur-puppet/pull/90 fixes as far as I know.

With that said, your issue may still be completely valid as multiple compilation masters are a rather unusual topology for our test cases. I would love it if you can reply back with results from using that new version so that we can be sure it's an issue that still needs fixing. Since we are currently working on support for deferred lookups https://puppet.com/docs/puppet/6.17/deferring_functions.html there is a high likelihood that it will obsolete the need for this certificate encryption completely in v3+ of this module.

PS: If the issue is confirmed with v2.0.4+ and if you are one of our corporate clients, it would also be highly advisable to create a Salesforce enhancement request for this as well so that you can asynchronously track our progress on this and bump up the priority of this work for us too.

Thanks, Srdjan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-665097886, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN7547MXBTSPCEAG47CLDR53S37ANCNFSM4PHAXPHA .

[sgnn7]

@frankmundt That's great to hear 👍 . Yes, keep us updated please as we would like to know if there are things to fix! :)

[frankmundt]

Another update. Still having issues when enabling the Puppet Compile masters. Verified that the encrypted_token Puppet Fact is encrypted with the master certificate and not the node certificate. Installed the latest puppet agent - 6.15.0 and received the same results.

On Wed, Jul 29, 2020 at 10:53 AM Srdjan Grubor notifications@github.com wrote:

@frankmundt https://github.com/frankmundt That's great to hear 👍 . Yes, keep us updated please as we would like to know if there are things to fix! :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-665747350, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN7563RMP3E7ZJFYGNE7TR6BAWVANCNFSM4PHAXPHA .

[sgnn7]

@frankmundt Thank you for the update!

We are working on the Deferred spike issue right now that should solve this for you depending on the usage scenario (you can find it at https://github.com/cyberark/conjur-puppet/issues/104 if you want to add any info or suggestions) so do you mind letting us know just a couple of more things so that we can prioritize it properly:

• What configuration type are you using for the Conjur variables (Hiera config, on-machine config, embedded-in-manifest config, or some combination of the 3)?
• What Conjur auth method are you using (host factory tokens, host/api key on agent, host/api key on puppet master)?

Thanks, Srdjan

[frankmundt]

Wow - thanks for working on this so quickly. We are using hiera for the Conjur variables and using host factory tokens for authentication.

On Fri, Jul 31, 2020 at 11:03 AM Srdjan Grubor notifications@github.com wrote:

@frankmundt https://github.com/frankmundt Thank you for the update!

We are working on the Deferred spike issue right now that should solve this for you depending on the usage scenario (you can find it at #104 https://github.com/cyberark/conjur-puppet/issues/104 if you want to add any info or suggestions) so do you mind letting us know just a couple of more things so that we can prioritize it properly:

• What configuration type are you using for the Conjur variables (Hiera config, on-machine config, embedded-in-manifest config, or some combination of the 3)?
• What Conjur auth method are you using (host factory tokens, host/api key on agent, host/api key on puppet master)?

Thanks, Srdjan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-667199944, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN75436B3CJSQP7BHY6I3R6LTN7ANCNFSM4PHAXPHA .

[sgnn7]

Hey @frankmundt, HFTs are a bit trickier (vs host/apikey) as we're thinking the host creation may need to be done server-side at least on the initial authentication and Deferred seems to bypass that whole setup so we need some sort of way to flip between them if we can and that's currently a big architectural unknown for now. For initial PoC there is way too many logic paths to grapple all at the same time so we're probably going to try to support on-machine credentials with Deferred which will then guide us in how to approach the HFT side of that, if that makes sense. The other issue I linked in the previous comment should track that work and then we can see how to bolt HFTs onto that and I'll also try to keep this tread up to date as well.

Thanks, Srdjan

[frankmundt]

I think it makes sense. FYI - we use a custom function that calls the DAP to obtain the host factory token and then uses the Conjur class to establish the node identity. This gives us the ability to rotate the HFT if needed.

On Fri, Jul 31, 2020 at 1:13 PM Srdjan Grubor notifications@github.com wrote:

Hey @frankmundt https://github.com/frankmundt, HFTs are a bit trickier (vs host/apikey) as we're thinking the host creation may need to be done server-side at least on the initial authentication and Deferred seems to bypass that whole setup so we need some sort of way to flip between them if we can and that's currently a big architectural unknown for now. For initial PoC there is way too many logic paths to grapple all at the same time so we're probably going to try to support on-machine credentials with Deferred which will then guide us in how to approach the HFT side of that, if that makes sense. The other issue I linked in the previous comment should track that work and then we can see how to bolt HFTs onto that and I'll also try to keep this tread up to date as well.

Thanks, Srdjan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cyberark/conjur-puppet/issues/164#issuecomment-667273086, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSN752UI4WZ3WT5OX222FTR6MCVBANCNFSM4PHAXPHA .

[sgnn7]

Released with v3.0.0rc3 and other comments were branched out to other issues. Closing.

[frankmundt]

Nice work - thank you!