search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.51k stars 772 forks source link

API server not working for localhost/127.0.0.1 or any other alt IP/DNS #2710

Closed RafaelMoreira1180778 closed 1 year ago

RafaelMoreira1180778 commented 3 years ago

I am trying to perform the following operation: $ curl -H "Authorization Bearer: $TOKEN" -X DELETE "https://10.194.143.237:16443/api/v1/nodes/node-to-be-removed" where 10.194.143.237 is the IP of the master node on my multipass 4 VM cluster, all with microk8s installed.

I get the following error:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Running curl with the -k flag throws the following error: curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1).

By digging a little deeper I found the following (all commands ran on the master node, aka 10.194.143.237):

$ microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 10.194.143.214:19001 10.194.143.174:19001 10.194.143.237:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    ingress              # Ingress controller for external access
    openebs              # OpenEBS is the open-source storage solution for Kubernetes
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    dashboard            # The Kubernetes dashboard
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    kata                 # Kata Containers is a secure runtime with lightweight VMS
    keda                 # Kubernetes-based Event Driven Autoscaling
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    openfaas             # openfaas serverless framework
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    storage              # Storage class; allocates storage from host directory
    traefik              # traefik Ingress controller for external access
$ nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2021-11-04 15:46 WET
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000090s latency).
Not shown: 995 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
111/tcp  open  rpcbind
443/tcp  open  https
9099/tcp open  unknown
$ sudo systemctl status snap.microk8s.daemon-apiserver.service
● snap.microk8s.daemon-apiserver.service - Service for snap application microk8s.daemon-apiserver
     Loaded: loaded (/etc/systemd/system/snap.microk8s.daemon-apiserver.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2021-11-04 12:43:18 WET; 3h 33min ago
   Main PID: 24086 (code=exited, status=0/SUCCESS)

Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + ARCH=x86_64
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + export LD_LIBRARY_PATH=:/snap/microk8s/2551/lib:/snap/microk8s/2551/usr/lib:/snap/microk8s/2551/lib/x86_64-linux-gnu>
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + LD_LIBRARY_PATH=:/snap/microk8s/2551/lib:/snap/microk8s/2551/usr/lib:/snap/microk8s/2551/lib/x86_64-linux-gnu:/snap/>
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/2551/lib:/s>
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void::/snap/microk8s/2551/lib:/snap/mic>
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + '[' -e /var/snap/microk8s/2551/var/lock/lite.lock ']'
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + echo 'Will not run along with kubelite'
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: Will not run along with kubelite
Nov 04 12:43:18 stimulated-redstart microk8s.daemon-apiserver[24086]: + exit 0
Nov 04 12:43:18 stimulated-redstart systemd[1]: snap.microk8s.daemon-apiserver.service: Succeeded.

From what I can see (not sure about it) but the ports regarding the API are not open therefore the curl will not work, this is what I expected (from the microk8s slack):

$ curl https://127.0.0.1:16443/api/v1/nodes/my-node -X DELETE  -k -H "Authorization: Bearer $TOKEN" 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Success",
  "details": {
    "name": "my-node",
    "kind": "nodes",
    "uid": "4811f6ac-33b1-4688-baea-75388f00d15e"
  }
}

At first I tought that the problem might be in the request itself, being a DELETE of a Node but the symptoms occur on a simple get of ANY namespace for pods, nodes, anything really.

Is it a problem/configuration on my side? More information below:

$ curl --version
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

microk8s inspect: inspection-report-20211104_161154.tar.gz

ktsakalozos commented 3 years ago

Hi @RafaelMoreira1180778 can you try something like 

export SNAP_DATA="/var/snap/microk8s/current/"
curl -L --cert ${SNAP_DATA}/certs/server.crt --key ${SNAP_DATA}/certs/server.key --cacert ${SNAP_DATA}/certs/ca.crt https://127.0.0.1:16443/readyz

The service you should be looking at is snap.microk8s.daemon-kubelite, this is where all k8s services start from.

RafaelMoreira1180778 commented 3 years ago 
export SNAP_DATA="/var/snap/microk8s/current/"
curl -L --cert ${SNAP_DATA}/certs/server.crt --key ${SNAP_DATA}/certs/server.key --cacert ${SNAP_DATA}/certs/ca.crt https://127.0.0.1:16443/readyz

Hi @ktsakalozos, am I correct to assume that the default certificates installed on the machine are being used by curl and therefore we need to specify the certificate, the key and the certificate authority for a curl to work with microk8s/kubernetes API?

The service you should be looking at is snap.microk8s.daemon-kubelite, this is where all k8s services start from.

Thanks very much, will help me debug microk8s further down the road for sure!

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.