Fresh deployment fails with TLS error (no PEM block found in ca_private_key_pem) when using a self-signed CA certificate

[ninadpage]

I have created a self-signed certificate as root CA using openssl req -newkey rsa:2048 -nodes -keyout ca.key -x509 -days 365 -out ca.crt. The verification of both the key (using openssl rsa -in ca.key -check) and certificate (using openssl x509 -text -noout -in ca.crt) succeeds.

I'm trying to deploy a new Kubernetes cluster using this, with above key/cert as CA. I'm using tectonic_1.7.9-tectonic.1 which comes with Terraform v0.10.7. My terraform.tfvars is as follows:

// (optional) Extra AWS tags to be applied to created autoscaling group resources.
// This is a list of maps having the keys `key`, `value` and `propagate_at_launch`.
// 
// Example: `[ { key = "foo", value = "bar", propagate_at_launch = true } ]`
// tectonic_autoscaling_group_extra_tags = ""

// (optional) Unique name under which the Amazon S3 bucket will be created. Bucket name must start with a lower case name and is limited to 63 characters.
// The Tectonic Installer uses the bucket to store tectonic assets and kubeconfig.
// If name is not provided the installer will construct the name using "tectonic_cluster_name", current AWS region and "tectonic_base_domain"
// tectonic_aws_assets_s3_bucket_name = ""

// Instance size for the etcd node(s). Example: `t2.medium`. Read the [etcd recommended hardware](https://coreos.com/etcd/docs/latest/op-guide/hardware.html) guide for best performance
tectonic_aws_etcd_ec2_type = "t2.medium"

// (optional) List of additional security group IDs for etcd nodes.
// 
// Example: `["sg-51530134", "sg-b253d7cc"]`
// tectonic_aws_etcd_extra_sg_ids = ""

// The amount of provisioned IOPS for the root block device of etcd nodes.
// Ignored if the volume type is not io1.
tectonic_aws_etcd_root_volume_iops = "100"

// The size of the volume in gigabytes for the root block device of etcd nodes.
tectonic_aws_etcd_root_volume_size = "30"

// The type of volume for the root block device of etcd nodes.
tectonic_aws_etcd_root_volume_type = "gp2"

// (optional) List of subnet IDs within an existing VPC to deploy master nodes into.
// Required to use an existing VPC and the list must match the AZ count.
// 
// Example: `["subnet-111111", "subnet-222222", "subnet-333333"]`
// tectonic_aws_external_master_subnet_ids = ""

// (optional) If set, the given Route53 zone ID will be used as the internal (private) zone.
// This zone will be used to create etcd DNS records as well as internal API and internal Ingress records.
// If set, no additional private zone will be created.
// 
// Example: `"Z1ILINNUJGTAO1"`
// tectonic_aws_external_private_zone = ""

// (optional) ID of an existing VPC to launch nodes into.
// If unset a new VPC is created.
// 
// Example: `vpc-123456`
// tectonic_aws_external_vpc_id = ""

// (optional) List of subnet IDs within an existing VPC to deploy worker nodes into.
// Required to use an existing VPC and the list must match the AZ count.
// 
// Example: `["subnet-111111", "subnet-222222", "subnet-333333"]`
// tectonic_aws_external_worker_subnet_ids = ""

// (optional) Extra AWS tags to be applied to created resources.
// tectonic_aws_extra_tags = ""

// (optional) This configures master availability zones and their corresponding subnet CIDRs directly.
// 
// Example:
// `{ eu-west-1a = "10.0.0.0/20", eu-west-1b = "10.0.16.0/20" }`
// tectonic_aws_master_custom_subnets = ""

// Instance size for the master node(s). Example: `t2.medium`.
tectonic_aws_master_ec2_type = "t2.medium"

// (optional) List of additional security group IDs for master nodes.
// 
// Example: `["sg-51530134", "sg-b253d7cc"]`
// tectonic_aws_master_extra_sg_ids = ""

// (optional) Name of IAM role to use for the instance profiles of master nodes.
// The name is also the last part of a role's ARN.
// 
// Example:
//  * Role ARN  = arn:aws:iam::123456789012:role/tectonic-installer
//  * Role Name = tectonic-installer
// tectonic_aws_master_iam_role_name = ""

// The amount of provisioned IOPS for the root block device of master nodes.
// Ignored if the volume type is not io1.
tectonic_aws_master_root_volume_iops = "100"

// The size of the volume in gigabytes for the root block device of master nodes.
tectonic_aws_master_root_volume_size = "30"

// The type of volume for the root block device of master nodes.
tectonic_aws_master_root_volume_type = "gp2"

// (optional) If set to true, create private-facing ingress resources (ELB, A-records).
// If set to false, no private-facing ingress resources will be provisioned and all DNS records will be created in the public Route53 zone.
// tectonic_aws_private_endpoints = true

// (optional) If set to true, create public-facing ingress resources (ELB, A-records).
// If set to false, no public-facing ingress resources will be created.
// tectonic_aws_public_endpoints = true

// The target AWS region for the cluster.
tectonic_aws_region = "eu-west-1"

// Name of an SSH key located within the AWS region. Example: coreos-user.
tectonic_aws_ssh_key = "<redacted>"

// Block of IP addresses used by the VPC.
// This should not overlap with any other networks, such as a private datacenter connected via Direct Connect.
tectonic_aws_vpc_cidr_block = "10.0.0.0/16"

// (optional) This configures worker availability zones and their corresponding subnet CIDRs directly.
// 
// Example: `{ eu-west-1a = "10.0.64.0/20", eu-west-1b = "10.0.80.0/20" }`
// tectonic_aws_worker_custom_subnets = ""

// Instance size for the worker node(s). Example: `t2.medium`.
tectonic_aws_worker_ec2_type = "t2.medium"

// (optional) List of additional security group IDs for worker nodes.
// 
// Example: `["sg-51530134", "sg-b253d7cc"]`
// tectonic_aws_worker_extra_sg_ids = ""

// (optional) Name of IAM role to use for the instance profiles of worker nodes.
// The name is also the last part of a role's ARN.
// 
// Example:
//  * Role ARN  = arn:aws:iam::123456789012:role/tectonic-installer
//  * Role Name = tectonic-installer
// tectonic_aws_worker_iam_role_name = ""

// (optional) List of ELBs to attach all worker instances to.
// This is useful for exposing NodePort services via load-balancers managed separately from the cluster.
// 
// Example:
//  * `["ingress-nginx"]`
// tectonic_aws_worker_load_balancers = ""

// The amount of provisioned IOPS for the root block device of worker nodes.
// Ignored if the volume type is not io1.
tectonic_aws_worker_root_volume_iops = "100"

// The size of the volume in gigabytes for the root block device of worker nodes.
tectonic_aws_worker_root_volume_size = "30"

// The type of volume for the root block device of worker nodes.
tectonic_aws_worker_root_volume_type = "gp2"

// The base DNS domain of the cluster. It must NOT contain a trailing period. Some
// DNS providers will automatically add this if necessary.
// 
// Example: `openstack.dev.coreos.systems`.
// 
// Note: This field MUST be set manually prior to creating the cluster.
// This applies only to cloud platforms.
// 
// [Azure-specific NOTE]
// To use Azure-provided DNS, `tectonic_base_domain` should be set to `""`
// If using DNS records, ensure that `tectonic_base_domain` is set to a properly configured external DNS zone.
// Instructions for configuring delegated domains for Azure DNS can be found here: https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns
tectonic_base_domain = "<redacted>"

// (optional) The content of the PEM-encoded CA certificate, used to generate Tectonic Console's server certificate.
// If left blank, a CA certificate will be automatically generated.
tectonic_ca_cert = "path/to/ca.crt"

// (optional) The content of the PEM-encoded CA key, used to generate Tectonic Console's server certificate.
// This field is mandatory if `tectonic_ca_cert` is set.
tectonic_ca_key = "path/to/ca.key"

// (optional) The algorithm used to generate tectonic_ca_key.
// The default value is currently recommended.
// This field is mandatory if `tectonic_ca_cert` is set.
tectonic_ca_key_alg = "RSA"

// (optional) This declares the IP range to assign Kubernetes pod IPs in CIDR notation.
// tectonic_cluster_cidr = "10.2.0.0/16"

// The name of the cluster.
// If used in a cloud-environment, this will be prepended to `tectonic_base_domain` resulting in the URL to the Tectonic console.
// 
// Note: This field MUST be set manually prior to creating the cluster.
// Warning: Special characters in the name like '.' may cause errors on OpenStack platforms due to resource name constraints.
tectonic_cluster_name = "<redacted>"

// (optional) The Container Linux update channel.
// 
// Examples: `stable`, `beta`, `alpha`
// tectonic_container_linux_channel = "stable"

// The Container Linux version to use. Set to `latest` to select the latest available version for the selected update channel.
// 
// Examples: `latest`, `1465.6.0`
tectonic_container_linux_version = "latest"

// (optional) This only applies if you use the modules/dns/ddns module.
// 
// Specifies the RFC2136 Dynamic DNS server key algorithm.
// tectonic_ddns_key_algorithm = ""

// (optional) This only applies if you use the modules/dns/ddns module.
// 
// Specifies the RFC2136 Dynamic DNS server key name.
// tectonic_ddns_key_name = ""

// (optional) This only applies if you use the modules/dns/ddns module.
// 
// Specifies the RFC2136 Dynamic DNS server key secret.
// tectonic_ddns_key_secret = ""

// (optional) This only applies if you use the modules/dns/ddns module.
// 
// Specifies the RFC2136 Dynamic DNS server IP/host to register IP addresses to.
// tectonic_ddns_server = ""

// (optional) DNS prefix used to construct the console and API server endpoints.
// tectonic_dns_name = ""

// (optional) The path of the file containing the CA certificate for TLS communication with etcd.
// 
// Note: This works only when used in conjunction with an external etcd cluster.
// If set, the variable `tectonic_etcd_servers` must also be set.
// tectonic_etcd_ca_cert_path = "/dev/null"

// (optional) The path of the file containing the client certificate for TLS communication with etcd.
// 
// Note: This works only when used in conjunction with an external etcd cluster.
// If set, the variables `tectonic_etcd_servers`, `tectonic_etcd_ca_cert_path`, and `tectonic_etcd_client_key_path` must also be set.
// tectonic_etcd_client_cert_path = "/dev/null"

// (optional) The path of the file containing the client key for TLS communication with etcd.
// 
// Note: This works only when used in conjunction with an external etcd cluster.
// If set, the variables `tectonic_etcd_servers`, `tectonic_etcd_ca_cert_path`, and `tectonic_etcd_client_cert_path` must also be set.
// tectonic_etcd_client_key_path = "/dev/null"

// The number of etcd nodes to be created.
// If set to zero, the count of etcd nodes will be determined automatically.
// 
// Note: This is not supported on bare metal.
tectonic_etcd_count = "0"

// (optional) List of external etcd v3 servers to connect with (hostnames/IPs only).
// Needs to be set if using an external etcd cluster.
// Note: If this variable is defined, the installer will not create self-signed certs.
// To provide a CA certificate to trust the etcd servers, set "tectonic_etcd_ca_cert_path".
// 
// Example: `["etcd1", "etcd2", "etcd3"]`
// tectonic_etcd_servers = ""

// (optional) If set to `true`, all etcd endpoints will be configured to use the "https" scheme.
// 
// Note: If `tectonic_experimental` is set to `true` this variable has no effect, because the experimental self-hosted etcd always uses TLS.
// tectonic_etcd_tls_enabled = true

// If set to true, experimental Tectonic assets are being deployed.
tectonic_experimental = false

// The path to the tectonic licence file.
// You can download the Tectonic license file from your Account overview page at [1].
// 
// [1] https://account.coreos.com/overview
// 
// Note: This field MUST be set manually prior to creating the cluster unless `tectonic_vanilla_k8s` is set to `true`.
tectonic_license_path = "path/to/tectonic-license.txt"

// The number of master nodes to be created.
// This applies only to cloud platforms.
tectonic_master_count = "1"

// (optional) Configures the network to be used in Tectonic. One of the following values can be used:
// 
// - "flannel": enables overlay networking only. This is implemented by flannel using VXLAN.
// 
// - "canal": [ALPHA] enables overlay networking including network policy. Overlay is implemented by flannel using VXLAN. Network policy is implemented by Calico.
// 
// - "calico": [ALPHA] enables BGP based networking. Routing and network policy is implemented by Calico. Note this has been tested on baremetal installations only.
// tectonic_networking = "flannel"

// The path the pull secret file in JSON format.
// This is known to be a "Docker pull secret" as produced by the docker login [1] command.
// A sample JSON content is shown in [2].
// You can download the pull secret from your Account overview page at [3].
// 
// [1] https://docs.docker.com/engine/reference/commandline/login/
// 
// [2] https://coreos.com/os/docs/latest/registry-authentication.html#manual-registry-auth-setup
// 
// [3] https://account.coreos.com/overview
// 
// Note: This field MUST be set manually prior to creating the cluster unless `tectonic_vanilla_k8s` is set to `true`.
tectonic_pull_secret_path = "path/to/config.json"

// (optional) This declares the IP range to assign Kubernetes service cluster IPs in CIDR notation.
// The maximum size of this IP range is /12
// tectonic_service_cidr = "10.3.0.0/16"

// If set to true, a vanilla Kubernetes cluster will be deployed, omitting any Tectonic assets.
tectonic_vanilla_k8s = false

// The number of worker nodes to be created.
// This applies only to cloud platforms.
tectonic_worker_count = "2"

But the terraform apply command fails with following errors:

Error applying plan:

5 error(s) occurred:

* module.identity_certs.tls_locally_signed_cert.identity_server: 1 error(s) occurred:

* tls_locally_signed_cert.identity_server: no PEM block found in ca_private_key_pem
* module.kube_certs.tls_locally_signed_cert.apiserver: 1 error(s) occurred:

* tls_locally_signed_cert.apiserver: no PEM block found in ca_private_key_pem
* module.identity_certs.tls_locally_signed_cert.identity_client: 1 error(s) occurred:

* tls_locally_signed_cert.identity_client: no PEM block found in ca_private_key_pem
* module.ingress_certs.tls_locally_signed_cert.ingress: 1 error(s) occurred:

* tls_locally_signed_cert.ingress: no PEM block found in ca_private_key_pem
* module.kube_certs.tls_locally_signed_cert.kubelet: 1 error(s) occurred:

* tls_locally_signed_cert.kubelet: no PEM block found in ca_private_key_pem

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

I also tried with -refresh=false without any effect (link).

EDIT: Updated config to add tectonic_ca_key_alg = "RSA" parameter which does not fix the issue.

[squat]

@ninadpage I believe the issue is that you omitted this configuration parameter:

// (optional) The algorithm used to generate tectonic_ca_key.
// The default value is currently recommended.
// This field is mandatory if `tectonic_ca_cert` is set.
// tectonic_ca_key_alg = "RSA"

Without this, the terraform cannot generate new certificates based on your ca

[ninadpage]

@squat Thank you for catching that!

Unfortunately, it does not fix the issue: terraform apply still fails with similar error.

[hhoover]

Don't use the path to the file, try using the actual content of the CA key and CA cert, one line each:

tectonic_ca_cert = "---BEGIN..."
tectonic_ca_key = "---BEGIN..."

[squat]

@hhoover yes, you are completely right. @ninadpage please try changing the certs/keys from file paths to the actual PEM content. Closing this for now. Please re-open if this you continue to run into this issue.