Manually verify that the Conjur Puppet plugin does not work in Windows with Puppet v6

Overview

We haven't validated that the certificate bug in #44 impacts both Windows and Linux users with Puppet v6. In this card, we'd like to:

Validate that the bug #44 also impacts Windows environments when used with Puppet v6
Document the workflow used to validate the bug in Windows, so that others who are also working in Windows will have some reference notes to get started

Definition of Done

Test environment:

Windows Server 2012 or 2016
Puppet v6 (clients and server)
Conjur or DAP (either way is fine - just document what you used) configured with a single Conjur host identity for the Puppet master server and nodes
The latest release of our Puppet integration

Using this environment, validate that you can run through a scenario like the smoketest example - ie that you can configure Puppet with a Conjur host and API key and successfully use Puppet to retrieve a secret value.

If you cannot, document the errors that you see along the way and any suggestions / ideas you have for how to address them (if any).

AC:

[ ] There are notes on this issue about the errors encountered when running through the scenario outlined above
[ ] There are notes on this issue about how the test environment was created, so that others may try to follow the same instructions and try to reproduce

Note: if you don't have access to a Windows 12 or 16 server environment, any env you can find that will validate this flow on Windows would do - please document what you end up using.

As I understand the solution we need will include multiple hosts where each API key will serve different puppet role

@Tovli in #20 we are only supporting the existing (single host / API key or host factory) authentication modes. to make this card simple, we asked to use the single host / API key mode.

puppet-master latest version puppetserver on ubuntu + conjur module

puppet-agent latest puppet-agent + conjur module os: windows server 2012

puppetmaster configuration /ete/hostname set to the private dns of the ec2 /etc/hosts map ip to puppet(otherwise problems connecting to puppetserver locally)

puppet-agnet configuration "C:\ProgramData\PuppetLabs\puppet\etc\puppet.conf"

[main]
certname = puppet-agent
server = ip-172-31-34-248.eu-central-1.compute.internal **same as server hostname**
autoflush = true
manage_internal_file_permissions = false
environment = production

server node definition /etc/puppetlabs/code/environments/production/manifests/site.pp

node puppet-agent {
  include conjur
}

running on agent: puppet agent -t yields the following error

C:\Users\Administrator> puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::conjur for puppet-agent (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 2, column: 3) on node puppet-agent
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

When I've installed conjur module on the master the error was gone. but I got an error related to unix/windows path of files Error: Failed to apply catalog: Parameter path failed on File[/etc/conjur.conf]: File paths must be fully qualified, not '/etc/conjur.conf' (file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/config_files.pp, line: 12)

Which is related to:

if $facts['os']['family'] == 'Windows' {
    require conjur::config::registry
    require conjur::identity::wincred
  } else {
    require conjur::config::files
    require conjur::identity::files
  }

in class conjur init.pp of conjur module

Current progress checkpoint:

Got Puppet OSS server v6 installed on an Ubuntu VM using the scripts here
- Fully automated script - just needs sudo on machine
- Script is flexible - can do v5 or v6 with a var change
- It auto-signs agent CSRs
- It downloads/installs conjur-puppet git-master plugin
- Doesn't have any puppet config (*.pp) scripts yet
Got an agent able to converge from that master using the script in the same link:
- Ubuntu agent
- Docker-based
- Unique hostname
- Waits for cert if server is set to not auto-sign CSRs
Didn't get far with Windows Agent due to VM issues but the docs are here for it.

(All of the configuraiton feom my previous comment applies here as well) puppet master and sever installed on ubuntu puppet agent installed on windows server 2012

following the README in this repo I have set all the reg keys: https://github.com/cyberark/conjur-puppet#pre-establish-host-identity-on-windows-hosts

[HKEY_LOCAL_MACHINE\SOFTWARE\CyberArk\Conjur]
"ApplianceUrl"="https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com"
"Version"=dword:00000005
"Account"="CyberArk"
"SslCertificate"=[cert]

Note reg ADD HKLM\Software\CyberArk\Conjur /v SslCertificate /t REG_SZ must be REG_MULTI_SZ

I have also set the Credentials for Conjur (host api key) in Windows Credential Manager.

Step 1 On the master I have created a role as described in the README On the master /etc/puppetlabs/code/environments/production/manifests/site.pp

node puppet-agent {
  include conjur
}

Step 1 Output

running puppet-agent -t on Windows yields the following output: Parameter 'appliance_url' expects a String value, got Undef

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Class[Conjur]: parameter 'appliance_url' expects a String value, got Undef (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 3, column: 3) on node puppet-agent
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Step 2 Add applicance_url to the role like so:

node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
  }
}

Note if the appliance_url is missing the protocol or ends with a / the catalog compilation will fail with un clear error

Step 2 Output puppet-agent -t on Windows yields the following output: parameter 'login' expects a String value, got Undef

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, 'conjur::token' parameter 'login' expects a String value, got Undef (file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/init.pp, line: 38, column: 35) on node puppet-agent
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Step 3 Add 'authn_login' the role like so:

node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1"
  }
}

Step 3 Output puppet-agent -t on Windows yields the following output: 'conjur::token' parameter 'key' expects a Sensitive value, got Undef

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, 'conjur::token' parameter 'key' expects a Sensitive value, got Undef (file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/init.pp, line: 38, column: 35) on node puppet-agent
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Step 4 Add 'authn_api_key' the role like so:

node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
   }
}

Step 4 Output puppet-agent -t on Windows yields the following output: certificate verify failed

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, certificate verify failed (file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/init.pp, line: 38, column: 35) on node puppet-agent
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Step 5 Add 'ssl_certificate' the role like so:

 node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
    ssl_certificate => file("/etc/conjur.pem"),
  }
}

Step 5 Output puppet-agent -t on Windows yields the following output on DAP: "/authn/users/host%2Fpuppetdemo%2Fpuppet_node1/authenticate"

<14>1 2020-06-30T08:14:02.000+00:00 a5afc077ee46 conjur-possum 1372 - [meta sequenceId="6"] [origin=172.31.34.248] [request_id=245de19c-41ac-4020-a121-0a41561ed5db] [tid=14483] Started POST "/authn/**users**/host%2Fpuppetdemo%2Fpuppet_node1/authenticate" for 172.31.34.248 at 2020-06-30 08:14:02 +0000

Version in regkey is ignored!

Step 6 Add 'version' the role like so:

node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
    ssl_certificate => file("/etc/conjur.pem"),
    version         => 5,
  }
}

Step 6 Output puppet-agent -t on Windows yields the following output: account is required for v5

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, **account** is required for v5 (file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/init.pp, line: 38, column: 35) on node puppet-agent
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Step 7 Add 'account' the role like so:

 node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
    ssl_certificate => file("/etc/conjur.pem"),
    version         => 5,
    account         => "CyberArk"
  }
}

Step 7 Output puppet-agent -t on Windows yields the following output: Failed to apply catalog: Parameter path failed on File[/etc/conjur.pem...file:config_files.pp

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Failed to apply catalog: Parameter path failed on File[/etc/conjur.pem]: File paths must be fully qualified, not '/etc/conjur.pem' (file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/config_files.pp, line: 4)

Step 8 Complete the role definition with secret retrieval and persistent

node puppet-agent {
  class { 'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
    ssl_certificate => file("/etc/conjur.pem"),
    version         => 5,
    account         => "CyberArk"
  }

 $secret = conjur::secret('puppetdemo/secretVar')

  notify {"Writing this secret to file: ${secret.unwrap}":}

  file { 'c:/tmp/test.pem':
    ensure  => file,
    content => conjur::secret('puppetdemo/secretVar'),
  }
}

Step 9 change paths in file: /etc/puppetlabs/code/environments/production/modules/conjur/manifests/config_files.pp to windows paths (e.g. etc/conjur.pem to c:/etc/conjur.pem)

class conjur::config_files inherits conjur {
  if $conjur::ssl_certificate {
    $cert_file = 'c:/etc/conjur.pem'
    file { $cert_file:
      replace => false,
      content => $conjur::ssl_certificate
    }
  } else {
    $cert_file = undef
  }

  file { **'c:/etc/conjur.conf'**:
    replace => false,
    content => conjur::config_yml(
      $conjur::appliance_url,
      $conjur::version,
      $conjur::authn_account,
      $cert_file
    )
  }

  if $conjur::api_key {
    file { '**c:/etc/conjur.identity':**
      replace   => false,
      mode      => '0400',
      backup    => false,
      show_diff => false,
      content   => conjur::netrc($conjur::client[uri], $conjur::authn_login, $conjur::api_key)
    }
  }
}

Final Output puppet-agent -t on Windows yields the following output: Writing this secret to file

C:\ProgramData\PuppetLabs\puppet\cache>puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for puppet-agent
Info: Applying configuration version '1593505920'
Notice: /Stage[main]/Conjur::Config_files/File[c:/etc/conjur.pem]/ensure: defined content as '{md5}5ad11bbafa2781e8a8afaceebf6a2d4d'
Notice: /Stage[main]/Conjur::Config_files/File[c:/etc/conjur.conf]/ensure: defined content as '{md5}8cde89c67789d824b04a3c655374a8ed'
Notice: /Stage[main]/Conjur::Config_files/File[c:/etc/conjur.identity]/ensure: changed [redacted] to [redacted]
Notice: Writing this secret to file: my secret
Notice: /Stage[main]/Main/Node[puppet-agent]/Notify[Writing this secret to file: my secret]/message: defined 'message' as 'Writing this secret to file: my secret'
Notice: /Stage[main]/Main/Node[puppet-agent]/File[c:/tmp/test.pem]/ensure: changed [redacted] to [redacted]
Notice: Applied catalog in 0.14 seconds

@eranha do you happen to know the versions of our puppet plugin that you used each time you tested this?

it looks like the first test was done with code that corresponded to the latest release (eg v2.0.3), and the second test was done with the code that's currently on master. can you please confirm? if that's the case, then I think this issue can be closed now.

@eranha do you happen to know the versions of our puppet plugin that you used each time you tested this?

it looks like the first test was done with code that corresponded to the latest release (eg v2.0.3), and the second test was done with the code that's currently on master. can you please confirm? if that's the case, then I think this issue can be closed now.

@izgeri I just ran puppet module install conjur-conjur if I run puppet module list on the master/agent I get:

C:/ProgramData/PuppetLabs/code/environments/production/modules
└── conjur-conjur (v1.2.0)
C:/ProgramData/PuppetLabs/code/modules (no modules installed)
C:/Program Files/Puppet Labs/Puppet/puppet/modules (no modules installed)

root@ip-172-31-34-248:/home/ubuntu# /opt/puppetlabs/puppet/bin/puppet module list
/etc/puppetlabs/code/environments/production/modules
└── conjur-conjur (v1.2.0)
/etc/puppetlabs/code/modules (no modules installed)
/opt/puppetlabs/puppet/modules (no modules installed)

I realize that this version is deprecated. I ran a quick test with the latest cyber-conjur puppet module version and got:

Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Unauthorized
Error: Failed to apply catalog: Parameter path failed on File[/tmp/test.pem]: File paths must be fully qualified, not '/tmp/test.pem' (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 15)

I've updated the path to c:/temp/test.pem and it worked. I don't know where the Unauthorized is coming from.

Note I've tried to use include conjur as as described in the README but it failed it works only with explicit parameters like so:

node puppet_agent  {
  class {'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
    ssl_certificate => file("/etc/conjur.pem"),
    version         => 5,
    account         => "CyberArk"
  }

 $secret = conjur::secret('puppetdemo/secretVar')

  notify {"Writing this secret to file: ${secret.unwrap}":}

  file { 'c:/tmp/test.pem':
    ensure  => file,
    content => conjur::secret('puppetdemo/secretVar'),
  }
}

@eranha this is surprising to me, actually. so you were running your original tests with 1.2.0 of our module, and the test from your last comment was run with 2.0.3 (right?) which does not include the bug fixes of the past week. based on this, I'm not sure I'm clear on when it worked and when it didn't. if you could put together a table (?) or something that summarizes the results you've seen, and success v errors, I think that could really help to clarify the state as you understand it.

I've confirmed that the unreleased conjur-puppet works on a v6 windows puppet agent talking to a v6 puppet server, using pre-provisioned machine identity via regedit and windows credential manager.

conjur-puppet module

The conjur-puppet module from this commit https://github.com/cyberark/conjur-puppet/commit/115c990edbd0615c22312bc6a236453db63de53c.

puppet server

v6 puppet sever.

The setup is based on this smoke_test setup https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/examples/puppetmaster/docker-compose.yml. I run ./smoketest_e2e.sh then expose both conjur and the puppet server on the public internet using ngrok.

Puppet agent

v6 puppet agent on windows
preprovisioned machine identity using regedit and windows credentials manager

I run it using this script https://gist.github.com/doodlesbykumbi/f77073f933355a804609922d7f18f8e3

Results

Success. The run of the agent can be seen at:

@eranha this is surprising to me, actually. so you were running your original tests with 1.2.0 of our module, and the test from your last comment was run with 2.0.3 (right?) which does not include the bug fixes of the past week. based on this, I'm not sure I'm clear on when it worked and when it didn't. if you could put together a table (?) or something that summarizes the results you've seen, and success v errors, I think that could really help to clarify the state as you understand it.

@izgeri The below puppet file (site.pp ) taken from the README on both versions failed

include conjur

The below puppet role (taken from our puppet repo examples)

node puppet_agent  {
  class {'conjur':
    appliance_url   => "https://ec2-18-194-148-1.eu-central-1.compute.amazonaws.com",
    authn_login     => "host/puppetdemo/puppet_node1",
    authn_api_key   => Sensitive("2167wn2h0ewzy30qy0za33kh4eb2t49p5s3ybxt162b23ygk2np02f1"),
    ssl_certificate => file("/etc/conjur.pem"),
    version         => 5,
    account         => "CyberArk"
  }

 $secret = conjur::secret('puppetdemo/secretVar')

  notify {"Writing this secret to file: ${secret.unwrap}":}

  file { 'c:/tmp/test.pem':
    ensure  => file,
    content => conjur::secret('puppetdemo/secretVar'),
  }
}

Failed in version 1.2.0 until I've changed the paths in config_params.pp file to windows paths. In Version 2.3.0 config_params.pp does not exist, it passed with no error except from the Unauthorised message in console.

@doodlesbykumbi Did you apply any role/pp file, using conjur's puppet module?

@eranha yes I applied the following role

File { backup => false }

node default {
  file { ‘c:/tmp/puppet-in-docker’:
    ensure  => present,
    content => ‘This file is for demonstration purposes only’,
  }

  if ($facts[‘conjur_smoke_test’]) {
    notify { “Including conjur module...”: }
    include conjur

    notify { “Grabbing ‘inventory/db-password’ secret...”: }
    $secret = conjur::secret(‘inventory/db-password’)

    notify { “Writing secret ‘${secret.unwrap}’ to c:/tmp/test.pem...”: }
    file { ‘c:/tmp/test.pem’:
      ensure  => file,
      content => conjur::secret(‘inventory/db-password’),
    }

    notify { “Done!”: }
  }
}

cyberark / conjur-puppet