activemq_keystores is not creating keystores

[sdodson]

Our QE team have a host that's not creating the keystores, it's deploying the CA, Certificate, and Key but for some reason it's not creating the keystore, no errors or anything. I thought perhaps the keystore and trust files existed and were timestamped sometime in the future, but that's not the case.

Notice: /Stage[main]/Openshift_origin::Update_conf_files/File[/etc/resolv.conf]/content: content changed '{md5}4932df750095b3e044ac7e1661cdc4aa' to '{md5}add67da47552676c820c73ea523eb75e'
Notice: /Stage[main]/Openshift_origin::Msgserver/Package[activemq]/ensure: created
Notice: /Stage[main]/Openshift_origin::Msgserver/Package[activemq-client]/ensure: created
Notice: /Stage[main]/Openshift_origin::Msgserver/File[/var/run/activemq/]/mode: mode changed '0755' to '0750'
Notice: /Stage[main]/Openshift_origin::Activemq_keystores/File[/etc/activemq/ssl_credentials]/ensure: created
Notice: /Stage[main]/Openshift_origin::Activemq_keystores/File[/etc/activemq/ssl_credentials/activemq_certificate.pem]/ensure: defined content as '{md5}a6b18f41ecc962da5b286b9c005aa6e4'
Notice: /Stage[main]/Openshift_origin::Activemq_keystores/File[/etc/activemq/ssl_credentials/activemq_private.pem]/ensure: defined content as '{md5}80492a6c6c03a0eae01d4de604151eae'
Notice: /Stage[main]/Openshift_origin::Activemq_keystores/File[/etc/activemq/ssl_credentials/ca.pem]/ensure: defined content as '{md5}e03744707a7309f6435dfd3f496b5a48'
Notice: /Stage[main]/Openshift_origin::Msgserver/File[activemq.xml config]/content:

This is originally from https://bugzilla.redhat.com/show_bug.cgi?id=1172261

[sdodson]

@andrewklau any idea?

[sdodson]

All of their tests are being done on a pristine system and I see signs that the pluginsync is failing. So right now I suspect this is happening because pluginsync hasn't sync'd java_ks. But I'm really surprised there aren't errors showing that it's trying to create resources for which it doesn't have a defined type for.

This is seen before the logs I've previously pasted.

Notice: /File[/var/opt/lib/pe-puppet/lib/puppet/parser/functions/postgresql_escape.rb]/ensure: defined content as '{md5}2e136fcd653ab38d831c5b40806d47d1'
Error: /File[/var/opt/lib/pe-puppet/lib/puppet/parser/functions/postgresql_password.rb]/ensure: change from absent to file failed: execution expired
Error: Could not retrieve plugin: execution expired
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_puppetdb_server_status.rb
Warning: Could not load fact file /var/opt/lib/pe-puppet/lib/facter/pe_puppetdb_server_status.rb: cannot load such file -- puppet/util/pe_puppetdb_validator
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/platform_tag.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_postgres_default_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/windows.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/postgres_default_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/staging_http_get.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/custom_auth_conf.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_build.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_version.rb
Info: Caching catalog for node1.openstacklocal
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
   (at /opt/puppet/lib/ruby/site_ruby/1.9.1/puppet/type.rb:816:in `set_default')
Info: Applying configuration version '1418181217'

[sdodson]

puppetd ignores resources in the catalog if the type code cannot be found https://projects.puppetlabs.com/issues/4415

Puppet run continues despite failed Pluginsync https://tickets.puppetlabs.com/browse/PUP-1763

[andrewklau]

Is pluginsync successful when msgserver_tls_enabled is set to false? pluginsync is required for the module to work properly, it could be the puppetmaster-client has not been setup properly to work with pluginsync.

I also noticed msgserver_tls_keystone_password is not being defined. By default it's password, we should probably add a check to make sure it's equal to or greater than 8 otherwise it'll also fail. I can't remember if it was a silent failure or it reported an error.

Otherwise, I'm not too sure.

[sdodson]

I'm pretty sure it's just a timeout issue. The host where they're running puppet agent is ~250ms from the puppet master. On the first puppet run it has to pull down all of the plugins and those hundreds of serialized requests exceed the default 120second timeout. If puppet were run again it would likely work as pluginsync is incremental. I've asked them to re-test with --configtimeout 9000

I'll close this once I have confirmation that works and we'll work on co-locating the puppet master with the test hosts to improve our test bed.

[sdodson]

not really an issue