Closed toshovski closed 1 year ago
Hi @toshovski
Chrome blocks self signed certificates. This might help you: https://github.com/ubuntu/microk8s/issues/945#issuecomment-593843714
Not all of them. the dashboard is signing a bad certificates. Is there a way to pass my own certificate by using micrk8s.enable dashboard?
This is how it looks like:
The certificate:
The page:
When I add a CN to the certificate, Google Chrome still allows me to proceed.
The certificate:
The page:
I could disable the certificate check as a workaround for now, but this is not a solution
Same here (microk8s 1.18), dashboard (v2.0.0-rc5) certificate is invalid even if I curl to the service locally. How did you manage to fill the common name and generate a new certificate?
$ kubectl -n kube-system describe service kubernetes-dashboard|grep Endpoints
Endpoints: 10.1.88.29:8443
curl https://10.1.88.29:8443
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
$ curl -kv https://10.1.88.29:8443
* Rebuilt URL to: https://10.1.88.29:8443/
* Trying 10.1.88.29...
* Connected to 10.1.88.29 (10.1.88.29) port 8443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
* common name: (does not match '10.1.88.29')
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC
* certificate version: #3
* subject:
* start date: Tue, 21 Apr 2020 11:11:09 GMT
* expire date: Wed, 21 Apr 2021 11:11:09 GMT
* issuer:
* compression: NULL
The problem is that the self-signed certificate does not use the subjectAltName
. Following the advices in https://stackoverflow.com/a/43666288/329263 I was able to generate a certificate with all the needed hostnames/IPs (cluster ip, localhost, LAN IP/name), replace the secret kubernetes-dashboard-certs
and specify it into the kubernetes-dashboard
configuration as advised in https://github.com/kubernetes/dashboard/blob/master/docs/user/certificate-management.md#self-signed-certificate. Finally I'm able to access the dashboard via the kube proxy.
./create_root_cert_and_key.sh
./create_certificate_for_domain.sh <LAN_HOSTNAME> <LAN_IP> localhost 127.0.0.1 <CLUSTER_IP>
mv <LAN_HOSTNAME>.crt dashboard.crt
kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=device.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
# add:
- args:
- --tls-cert-file=/dashboard.crt
- --tls-key-file=/device.key
openssl s_client -showcerts -connect <CLUSTER_IP>:443
kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443 --address 0.0.0.0
desktop$ open https://<LAN_HOSTNAME>
I am having this same problem. I had to use an old version of firefox to access the dashboard. I even added the cert to my macos keychain access and said trust, and chrome still didn't like it.
I tried running the scripts in comment https://github.com/ubuntu/microk8s/issues/1046#issuecomment-617190819, but it didn't generate a dashboard.crt, so I got stuck.
I installed microk8s yesterday, latest/edge channel, and I got the same certification error.
Changing Chrome to accept self-signed certificates doesn't help. Based on what I read in the #945 and in the PR #970 I still need to adjust certificates on the machine kubernetes is running.
For me, who doesn't like to deal with certificates it is a bad user experience facing with this especially with the promise of Zero-ops
kubernetes.
https://github.com/kubernetes/dashboard/issues/2995#issuecomment-551309479 This worked namespace may be different (kube-system)
This is blocking me as well
This might be related: https://www.techprowd.com/automatic-ssl-certificates-for-home-microk8s-setup-using-letsencrypt/
Also you may want to read the comments at https://news.ycombinator.com/item?id=24361930
Chrome Quick-Fix: Go to chrome://flags/#allow-insecure-localhost And enable "Allow invalid certificates for resources loaded from localhost."
Just installed microk8s dashboard on the latest snap --classic microk8s channel. Have the same problem. The dashboard cert is invalid and can't get any browser to use it. This needs to get fixed.
Same problem here. Installed microk8s
from the edge
-channel. Started the dashboard and cannot access it via Chrome. Even the Workaround with Allow invalid certificates for resources loaded from localhost.
is not allowing me to access the dashboard.
Here is a quick dump of the presented certificate made with the openssl s_client
:
openssl s_client -connect 10.152.183.176:443
CONNECTED(00000005)
depth=0
verify error:num=18:self signed certificate
verify return:1
depth=0
verify return:1
---
Certificate chain
0 s:
i:
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBADCBpqADAgECAhAXt3NvfFYj0jLAsyyR1Mf5MAoGCCqGSM49BAMCMAAwHhcN
MjAxMTI3MjEyMjU2WhcNMjExMTI3MjEyMjU2WjAAMFkwEwYHKoZIzj0CAQYIKoZI
zj0DAQcDQgAE+8efMOScjbJJdhofFE5JMcS8DWutAgaA/+OP1wbdt9WtlKH1ovYu
LjnYgM8KD4kuN1HbZ9avnYkiA2QvPmNpZaMCMAAwCgYIKoZIzj0EAwIDSQAwRgIh
ANyrJrxlOX3S/7cigOwCwCEj1cxFgvP+F5KD/MuXVYyKAiEA7Zr17umwbvfCmhtp
2M76xSHvI7FuSkWp+VDao41JwF4=
-----END CERTIFICATE-----
subject=
issuer=
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 615 bytes and written 380 bytes
Verification error: self signed certificate
So beside that the Cert is self signed, it is virtually empty? No subject, no issuer. At least it is valid for one year. :D
My microk8s was installed using --classic
and here's what works for me:
mkdir certs
openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt
kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
# modify section args as follows
args:
- --tls-cert-file=/dashboard.crt
- --tls-key-file=/dashboard.key
#- --auto-generate-certificates
My microk8s was installed using
--classic
and here's what works for me:mkdir certs openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*" openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt kubectl -n kube-system delete secret kubernetes-dashboard-certs kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml # modify section args as follows args: - --tls-cert-file=/dashboard.crt - --tls-key-file=/dashboard.key #- --auto-generate-certificates
This worked for me with some minor mods:
openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -out dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt
kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
# modify section args as follows
args:
- --tls-cert-file=dashboard.crt
- --tls-key-file=dashboard.key
#- --auto-generate-certificates
After stopping and restarting microk8s dashboard-proxy I can now log in using Chrome 88.
mkdir certs openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*" openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt kubectl -n kube-system delete secret kubernetes-dashboard-certs kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
What does the "modify section args" part mean? I'm not understanding what this is applied to.
mkdir certs openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*" openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt kubectl -n kube-system delete secret kubernetes-dashboard-certs kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
What does the "modify section args" part mean? I'm not understanding what this is applied to.
Do you mean this?
sudo microk8s dashboard-proxy --address 0.0.0.0 --tls-cert-file=dashboard.crt --tls-key-file=dashboard.key
Run these commands:
openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -out dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt
kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
The last command opens a yaml configuration in editor (e.g. vi). Look for the 'args' section and change it so it looks like this:
args:
- --tls-cert-file=dashboard.crt
- --tls-key-file=dashboard.key
#- --auto-generate-certificates
This comments out the auto-generate-certificates option and explicitly sets the key and cert file.
Then save and exit the editor to apply the changes.
If 'microk8s dashboard-proxy' is already running, press Control-C to stop it.
Then restart it with this command:
microk8s dashboard-proxy
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
Interesting. I'm getting timeouts on the second to last command. Trying to see how to get more information about the timeout details.
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
Unable to connect to the server: dial tcp 192.168.99.100:8443: i/o timeout
And the delete just hangs too (I didn't wait long enough for it to finish).
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
Interesting. I'm getting timeouts on the second to last command. Trying to see how to get more information about the timeout details.
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key Unable to connect to the server: dial tcp 192.168.99.100:8443: i/o timeout
And the delete just hangs too (I didn't wait long enough for it to finish).
So the only trick is to add microk8s in front of those commands. Seems obvious once you know.
microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
I managed to get this working after several gotchas...
Make sure dashboard-proxy is not running
Run these commands:
mkdir /tmp/certs
cd /tmp/certs
openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -out dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt
sudo microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
sudo microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
sudo microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
spec.template.spec.containers.args
:....
spec:
containers:
- args:
- --tls-cert-file=dashboard.crt
- --tls-key-file=dashboard.key
- --namespace=kube-system
....
Start dashboard-proxy
In Chrome 88, open console and long-press the reload button until a menu shows. Pick "Empty Cache and Hard Reload":
Another way I've just discovered to navigate around this issue (Tested this in Edge Chromium, but it should work in Chrome too)... If you click anywhere in the window and type 'thisisunsafe' it'll reload the page and navigate to the Kubernetes Dashboard login...
Another way I've just discovered to navigate around this issue (Tested this in Edge Chromium, but it should work in Chrome too)... If you click anywhere in the window and type 'thisisunsafe' it'll reload the page and navigate to the Kubernetes Dashboard login...
Yep, that will work in all Chromium-based browsers. But neither of these workarounds are ideal. I'm wondering, how come this isn't getting more attention? The bug has been up for well over a year. microk8s is such a great project, and it's disheartening for people who are just getting started to hit this kind of wall so early while following tutorials online.
The code to generate self-signed certificate above missing required extensions so that the browser see that the certificate is invalid (not valid for signing a server). For some people, it may happen to be valid because the openssl.cnf already contained predefined configuration which allow to create a valid SSL certificate.
To be sure, you can use these commands to generate a valid self-signed certificate:
mkdir ~/certs
cd ~/certs
tee openssl.cnf <<EOF
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
string_mask = utf8only
default_md = sha256
x509_extensions = server_cert
prompt = no
[ req_distinguished_name ]
commonName = localhost
[ server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
DNS.2 = *
EOF
# generate both key and certificate in one step
openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -new -x509 -days 3650 -out dashboard.crt -extensions server_cert -config openssl.cnf
Then update the certificate to k8s dashboard
microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
Update the dashboard config to use these certs:
microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
# edit spec.template.spec.containers.args as
....
spec:
containers:
- args:
- --tls-cert-file=dashboard.crt
- --tls-key-file=dashboard.key
- --namespace=kube-system
....
Restart the dashboard pod so that it recognizes the new certificate. In my case, the namespace contains dashboard pod is kube-system. It may be different in your case:
# look and see which namespace your pod is in
microk8s kubectl get pod --all-namespaces
# delete all dashboard pods from kube-system
microk8s kubectl delete -n kube-system $(microk8s kubectl get pod --all-namespaces -o name | grep dashboard)
Start the dashboard proxy again
microk8s dashboard-proxy
Now you will be able to see the bypass link when opening the dashboard in browser.
I'm writing a tutorial for our project, which will use microk8s, and hit this when documenting the dashboard for those new users to understand what is running. (I am almost entirely CLI .... )
Unfortunately this issue is a real impediment for the new user experience. I do empathise with the pain of creating self-signed certs. I struggled myself with it on egeria. I ended up with these -> https://github.com/odpi/egeria/tree/master/open-metadata-resources/open-metadata-deployment/certificates with a ref from https://odpi.github.io/egeria-docs/guides/admin/omag-server-platform-transport-level-security/#example-script-to-launch-egeria
I'm hitting the error in microk8s using safari on Monterey. There is no user bypass. As mentioned above, some of the certs need additional settings to allow safari/chrome to work at all. Of course this is only ever for demos, tutorials
The code to generate self-signed certificate above missing required extensions so that the browser see that the certificate is invalid (not valid for signing a server). For some people, it may happen to be valid because the openssl.cnf already contained predefined configuration which allow to create a valid SSL certificate.
To be sure, you can use these commands to generate a valid self-signed certificate:
mkdir ~/certs cd ~/certs tee openssl.cnf <<EOF [ req ] default_bits = 2048 distinguished_name = req_distinguished_name string_mask = utf8only default_md = sha256 x509_extensions = server_cert prompt = no [ req_distinguished_name ] commonName = localhost [ server_cert ] basicConstraints = CA:FALSE nsCertType = server nsComment = "OpenSSL Generated Server Certificate" subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer:always keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1 = localhost DNS.2 = * EOF # generate both key and certificate in one step openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -new -x509 -days 3650 -out dashboard.crt -extensions server_cert -config openssl.cnf
Then update the certificate to k8s dashboard
microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
Update the dashboard config to use these certs:
microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml # edit spec.template.spec.containers.args as .... spec: containers: - args: - --tls-cert-file=dashboard.crt - --tls-key-file=dashboard.key - --namespace=kube-system ....
Restart the dashboard pod so that it recognizes the new certificate. In my case, the namespace contains dashboard pod is kube-system. It may be different in your case:
# look and see which namespace your pod is in microk8s kubectl get pod --all-namespaces # delete all dashboard pods from kube-system microk8s kubectl delete -n kube-system $(microk8s kubectl get pod --all-namespaces -o name | grep dashboard)
Start the dashboard proxy again
microk8s dashboard-proxy
Now you will be able to see the bypass link when opening the dashboard in browser.
This solution work for me.
Could you implement this on new version of microk8s ?
Best regards
I've found this solution to renew the server certificates. works for me. after renewing all nodes are back alive: https://www.etissimo.de/blog/2022/04/03/renew-microk8s-kubernetes-server-certificates-annually/?lang=en
It would be easier if this feature was native in the product... right?
yes. it should be integrated into the product, also to avoid unpleasant system failures.
I made a simple and small (~3MB) docker tool to generate a self-signed certs. It is relatively simple to add multiple DNS domain names or specify good-looking basic information.
I hope it will help friends who need to make self-signed (generate a certificate that can be used by k8s):
docker run --rm -it -v `pwd`/certs:/ssl soulteary/certs-maker --FOR_K8S=on
If you need to add multiple domain names, just run:
docker run --rm -it -v `pwd`/certs:/ssl soulteary/certs-maker --FOR_K8S=on --CERT_DNS=apple.com,orange.com,pear.com
For more parameters and usage, you can refer to the project and code: https://github.com/soulteary/certs-maker
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
I'm afraid I have to disagree with this being closed as the community obviously thinks it's relevant.
The best would be if a certificate were generated automatically by microk8s as part of enabling the dashboard.
I would find it acceptable to update the documentation to help new users deal with the limitations.
When I execute microk8s.enable dashboard, is there a way to pass certificates? The current certificate is invalid and chrome doesn't allow me to access the dashboard.
I get the following error, and I cannot accept the risks anymore. On firefox I can still access it.