MountVolume.SetUp failed for volume "tls-certs" : secret "vpa-tls-certs" not found

[ghost]

When I created the VPA, there was no error, I created the VPA-admissions-Controller and found no VPA-TLs-Certs

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    <unknown>           default-scheduler  Successfully assigned kube-system/vpa-admission-controller-69c96bd8bd-vq5cr to mb
  Warning  FailedMount  60s (x8 over 2m3s)  kubelet, mb        MountVolume.SetUp failed for volume "tls-certs" : secret "vpa-tls-certs" not found
  Warning  FailedMount  0s                  kubelet, mb        Unable to attach or mount volumes: unmounted volumes=[tls-certs], unattached volumes=[vpa-admission-controller-token-c9d2p tls-certs]: timed out waiting for the condition
[root@ma hack]# kubectl  logs  vpa-admission-controller-69c96bd8bd-vq5cr -n kube-system
Error from server (BadRequest): container "admission-controller" in pod "vpa-admission-controller-69c96bd8bd-vq5cr" is waiting to start: ContainerCreating

[root@ma hack]# ./vpa-up.sh 
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created
serviceaccount/vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
serviceaccount/vpa-updater created
deployment.apps/vpa-updater created
serviceaccount/vpa-recommender created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus
...................................................................+++
.............+++
e is 65537 (0x10001)
unknown option -addext
req [options] <infile >outfile
where options  are
 -inform arg    input format - DER or PEM
 -outform arg   output format - DER or PEM
 -in arg        input file
 -out arg       output file
 -text          text form of request
 -pubkey        output public key
 -noout         do not output REQ
 -verify        verify signature on REQ
 -modulus       RSA modulus
 -nodes         don't encrypt the output key
 -engine e      use engine e, possibly a hardware device
 -subject       output the request's subject
 -passin        private key password source
 -key file      use the private key contained in file
 -keyform arg   key file format
 -keyout arg    file to send the key to
 -rand file:file:...
                load the file (or the files in the directory) into
                the random number generator
 -newkey rsa:bits generate a new RSA key of 'bits' in size
 -newkey dsa:file generate a new DSA key, parameters taken from CA in 'file'
 -newkey ec:file generate a new EC key, parameters taken from CA in 'file'
 -[digest]      Digest to sign with (see openssl dgst -h for list)
 -config file   request template file.
 -subj arg      set or modify request subject
 -multivalue-rdn enable support for multivalued RDNs
 -new           new request.
 -batch         do not ask anything during request generation
 -x509          output a x509 structure instead of a cert. req.
 -days          number of days a certificate generated by -x509 is valid for.
 -set_serial    serial number to use for a certificate generated by -x509.
 -newhdr        output "NEW" in the header lines
 -asn1-kludge   Output the 'request' in a format that is wrong but some CA's
                have been reported as requiring
 -extensions .. specify certificate extension section (override value in config file)
 -reqexts ..    specify request extension section (override value in config file)
 -utf8          input characters are UTF8 (default ASCII)
 -nameopt arg    - various certificate name options
 -reqopt arg    - various request text options

deployment.apps/vpa-admission-controller created
service/vpa-webhook created

[ghost]

Although the switch to the 0.8 branch comes, the unknown option-addext is still displayed when deploying

[root@ma vertical-pod-autoscaler]# git branch
  master
* remotes/origin/vpa-release-0.8
[root@ma vertical-pod-autoscaler]# ls
builder  common  deploy  e2e  examples  FAQ.md  go.mod  go.sum  hack  MIGRATE.md  OWNERS  pkg  README.md  vendor
[root@ma vertical-pod-autoscaler]# cd hack/
[root@ma hack]# ls
boilerplate.go.txt        deploy-for-e2e.sh  run-e2e-tests.sh   update-kubernetes-deps-in-e2e.sh  vpa-apply-upgrade.sh  vpa-process-yaml.sh   vpa-up.sh
convert-alpha-objects.sh  run-e2e.sh         update-codegen.sh  verify-codegen.sh                 vpa-down.sh           vpa-process-yamls.sh  warn-obsolete-vpa-objects.sh
[root@ma hack]# ./vpa-up.sh 
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created
serviceaccount/vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
serviceaccount/vpa-updater created
deployment.apps/vpa-updater created
serviceaccount/vpa-recommender created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus
......................................................................................................................................................+++
.....................................................................................................................+++
e is 65537 (0x10001)
unknown option -addext
req [options] <infile >outfile
where options  are

[bskiba]

This should not happen, the gencerts.sh script does not use -addext option in vpa-release-0.8 branch: https://github.com/kubernetes/autoscaler/blob/vpa-release-0.8/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh Can you verify the contents of gencerts.sh in your setup? $ cat pkg/admission-controller/gencerts.sh

[Woofenator]

Just out of curiosity, which openssl version supports the -addtext option?

[bskiba]

1.1.1 seems to have it: https://www.openssl.org/docs/man1.1.1/man1/req.html

[Woofenator]

Am on 1.1.1g on Mac, still throws error that option doesn't exist

[ghost]

How do I verify that? This is my file

[root@master vertical-pod-autoscaler]#  cat pkg/admission-controller/gencerts.sh
#!/bin/bash

# Copyright 2018 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Generates the a CA cert, a server key, and a server cert signed by the CA.
# reference:
# https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/webhook/gencerts.sh
set -e

CN_BASE="vpa_webhook"
TMP_DIR="/tmp/vpa-certs"

echo "Generating certs for the VPA Admission Controller in ${TMP_DIR}."
mkdir -p ${TMP_DIR}
cat > ${TMP_DIR}/server.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = DNS:vpa-webhook.kube-system.svc
EOF

# Create a certificate authority
openssl genrsa -out ${TMP_DIR}/caKey.pem 2048
openssl req -x509 -new -nodes -key ${TMP_DIR}/caKey.pem -days 100000 -out ${TMP_DIR}/caCert.pem -subj "/CN=${CN_BASE}_ca" -addext "subjectAltName = DNS:${CN_BASE}_ca"

# Create a server certiticate
openssl genrsa -out ${TMP_DIR}/serverKey.pem 2048
# Note the CN is the DNS name of the service of the webhook.
openssl req -new -key ${TMP_DIR}/serverKey.pem -out ${TMP_DIR}/server.csr -subj "/CN=vpa-webhook.kube-system.svc" -config ${TMP_DIR}/server.conf -addext "subjectAltName = DNS:vpa-webhook.kube-system.svc"
openssl x509 -req -in ${TMP_DIR}/server.csr -CA ${TMP_DIR}/caCert.pem -CAkey ${TMP_DIR}/caKey.pem -CAcreateserial -out ${TMP_DIR}/serverCert.pem -days 100000 -extensions SAN -extensions v3_req -extfile ${TMP_DIR}/server.conf

echo "Uploading certs to the cluster."
kubectl create secret --namespace=kube-system generic vpa-tls-certs --from-file=${TMP_DIR}/caKey.pem --from-file=${TMP_DIR}/caCert.pem --from-file=${TMP_DIR}/serverKey.pem --from-file=${TMP_DIR}/serverCert.pem

# Clean up after we're done.
echo "Deleting ${TMP_DIR}."
rm -rf ${TMP_DIR}

[bskiba]

This is the master version of the file, not from the vpa-release-0.8 branch

[mreferre]

I have a script that setup VPA (search for verticalpodautoscaler() in the file) that used to work just fine until a few weeks ago and it started to give me a black eye recently with the exact same error.

I am basically just cloning https://github.com/kubernetes/autoscaler.git and then running ./autoscaler/vertical-pod-autoscaler/hack/vpa-up.sh.

I am using Amazon Linux and my openssl version is (and has always been) this:

sh-4.2# openssl version  
OpenSSL 1.0.2k-fips  26 Jan 2017
sh-4.2#

[bskiba]

I recommend checking out vpa-release-0.8 branch instead of master in that case.

The change in master is needed as otherwise the webhook will not be able to install on k8s 1.20.X clusters.

[mreferre]

Thanks @bskiba. Out of curiosity, should I want to stick to master, what would I need to do? In other words what's breaking it? is it the openssl version? Thanks.

[ghost]

Okay, I've fixed the problem and just need to upgrade OpensSL to 1.1.1

kube-system   vpa-admission-controller-69c96bd8bd-4st7v   1/1     Running     0          39s
kube-system   vpa-recommender-765b6c5f59-rdxk4            1/1     Running     0          45s
kube-system   vpa-updater-86865896cf-z8bxn                1/1     Running     0          50s

[bskiba]

@mreferre yes, the openssl version you use needs to support -addext argument @zhaocheng173 Glad that it's working for you

[bskiba]

/close

[k8s-ci-robot]

@bskiba: Closing this issue.

In response to [this](https://github.com/kubernetes/autoscaler/issues/3397#issuecomment-670379602): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.