(maint) Add / consume pre-generated certs

[Iristyle]

(maint) Add pre-generated certs to shared gem

• In an effort to speed up the initial cluster startup time, prepare to prime volumes with certificates that are already generated.

  This includes certs for:

  • puppetserver CA
  • puppetserver (certname puppet.test)
  • bolt-server (certname pe-bolt-server.test)
  • console-services (certname pe-console-services.test)
  • orchestration-services (certname pe-orchestration-services.test)
  • postgres (certname postgres.test)
  • puppetdb (certname puppetdb.test)

• This does not yet include a pre-generated cert for:

  • puppet-agent (certname puppet-agent.test) This would require additional refactoring work to tests, but may be added at a later date

• NOTE: compose files currently use service name 'puppet', but the certs stored here are in a 'puppetserver' directory.

  This is because the 'puppet' service for pe-puppetserver has multiple VOLUMEs defined. Creating a helper method to install certs (in a later commit) will only ever copy cert files to the first VOLUME. It's easier to create a new dummy service that shares the correct VOLUME with the 'puppet' service to ensure certs are copied to the correct location.

(maint) Change compose domain to .test

• .internal was the previous domain in use by default for the docker-compose.yml

  In various other testing suites, that domain has been moved over to .test from .internal. The TLD test is recognized as a safe reserved TLD from RFC2606 (https://tools.ietf.org/html/rfc2606) that will never have an external DNS reservation

• The certs that were just added are based on .test, so align the Docker services around that name in preparation to use pre-generated certs

(maint) Fix teardown_container when stopped

• Stopped containers are not connected to a network.

  Detect a missing network value and don't attempt a disconnect when not necessary (which leads to an error message)

(maint) Add helper to preload certs into containers

• During a docker_compose_up() helper call, which all spec suites currently consume, add certs to containers IFF PRELOAD_CERTS environment variable is set to 1.

• docker_compose_up will now create but not start containers so that their volumes may be modified when the containers are not running

• The basic algorithm for installing certs is:

  • when a SERVICE is defined that matches directory certs/SERVICE
  • then copy into the sub-directory 'certs' for the first defined VOLUME for the given service using docker cp
  • any additional modifications to the permissions are expected to be done by the containers themselves

• To work under LCOW, which doesn't properly support docker cp, use a separate temp container and bind mount to the local filesystem. Have the transient container also share the desired VOLUME and use it to copy the files.

  The alternative of finding the location on disk in Windows was explored, but encountered permissions problems like:

Permission denied @ dir_s_mkdir - C:\ProgramData\docker\volumes\pupperware-commercial_bolt-server_data/certs

• As mentioned in prior commit, a special container will need to be defined named 'puppetserver' that shares the 'puppetserver' volume defined in the 'puppet' service. the 'puppet' service has 4 volumes, so there needs to be a way to uniquely identify the correct volume. The easiest thing to do is copy certs to the new service.

  puppetserver: image: alpine:3.10 volumes:

  this path is irrelevant as long as it exists

  • puppetserver:/opt/puppetlabs

• This can significantly improve startup times for the stack:

  Previously on OSX:

  Service puppet is healthy - running 76.8 seconds Service pe-bolt-server is healthy - running 93.9 seconds Service postgres is healthy - running 94.0 seconds Service puppetdb is healthy - running 212.7 seconds Service pe-console-services is healthy - running 218.8 seconds Service pe-orchestration-services is healthy - running 284.4 seconds

  After pre-loading certs on OSX:

  Service pe-bolt-server is healthy - running 10.9 seconds Service postgres is healthy - running 11.7 seconds Service puppet is healthy - running 53.1 seconds Service pe-console-services is healthy - running 155.2 seconds Service puppetdb is healthy - running 156.8 seconds Service pe-orchestration-services is healthy - running 195.7 seconds

(maint) Refactor to generic docker_volume_cp

• Create a new helper for copying local files to a volume, and use it on all platforms instead of just LCOW

• This method relies on a transient Alpine container for file copies

[nwolfe]

How are these going to be consumed?

Just drop them into the volume directories? Manually, or programmatically?

I like it!