microk8s stop command does not shutdown previous deployed containers

[muten84]

Summary

I've successfully installed microk8s with a basic default configuration. This is the output of microk8s status:

microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    traefik              # (community) traefik Ingress controller
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    storage              # (core) Alias to hostpath-storage add-on, deprecated
  disabled:
    argocd               # (community) Argo CD is a declarative continuous deployment for Kubernetes.
    cilium               # (community) SDN, fast with full network policy
    dashboard-ingress    # (community) Ingress definition for Kubernetes dashboard
    easyhaproxy          # (community) EasyHAProxy can configure HAProxy automatically based on ingress labels
    fluentd              # (community) Elasticsearch-Fluentd-Kibana logging and monitoring
    gopaddle-lite        # (community) Cheapest, fastest and simplest way to modernize your applications
    inaccel              # (community) Simplifying FPGA management in Kubernetes
    istio                # (community) Core Istio service mesh services
    jaeger               # (community) Kubernetes Jaeger operator with its simple config
    kata                 # (community) Kata Containers is a secure runtime with lightweight VMS
    keda                 # (community) Kubernetes-based Event Driven Autoscaling
    knative              # (community) Knative Serverless and Event Driven Applications
    kwasm                # (community) WebAssembly support for WasmEdge (Docker Wasm) and Spin (Azure AKS WASI)
    linkerd              # (community) Linkerd is a service mesh for Kubernetes and other frameworks
    multus               # (community) Multus CNI enables attaching multiple network interfaces to pods
    nfs                  # (community) NFS Server Provisioner
    ondat                # (community) Ondat is a software-defined, cloud native storage platform for Kubernetes.
    openebs              # (community) OpenEBS is the open-source storage solution for Kubernetes
    openfaas             # (community) OpenFaaS serverless framework
    osm-edge             # (community) osm-edge is a lightweight SMI compatible service mesh for the edge-computing.
    parking              # (community) Static webserver to park a domain. Works with EasyHAProxy.
    portainer            # (community) Portainer UI for your Kubernetes cluster
    shifu                # (community) Kubernetes native IoT software development framework.
    sosivio              # (community) Kubernetes Predictive Troubleshooting, Observability, and Resource Optimization
    trivy                # (community) Kubernetes-native security scanner
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000

I have on the same host a nginx server bound to port 80 and 443 with a proxy-pass configuration that forwards requests to an ingress LB that relies on Traefik. I deployed a service that starts a spring boot app written in java and when i run microk8s it starts all pods then when i try to reach the service from the external dns nginx use the proxy-pass directive and i can see the response using external dns. But when i run microk8s stop despite the expected behavior should be to stop all running pods i still can reach the spring boot app.

Furthermore i tried to invoke locally on the server the following commands:

wget http://127.0.0.1:31539/actuator/health
--2023-05-15 09:37:13--  http://127.0.0.1:31539/actuator/health
Connecting to 127.0.0.1:31539... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/vnd.spring-boot.actuator.v3+json]
Saving to: ‘health.1’

health.1                                               [ <=>                                                                                                             ]      49  --.-KB/s    in 0s      

2023-05-15 09:37:13 (4.35 MB/s) - ‘health.1’ saved [49]

Despit i run microk8s stop the service is still reacheble!!

**microk8s status
microk8s is not running, try microk8s start**
user@host:~/docker-registry$ wget http://127.0.0.1:31539/actuator/health
--2023-05-15 09:38:40--  http://127.0.0.1:31539/actuator/health
Connecting to 127.0.0.1:31539... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/vnd.spring-boot.actuator.v3+json]
Saving to: ‘health.4’

health.4                                               [ <=>                                                                                                             ]      49  --.-KB/s    in 0s      

2023-05-15 09:38:40 (3.79 MB/s) - ‘health.4’ saved [49]

The proxy-pass directive is enabled trough this configuration in nginx:

location /backend/ {
         proxy_set_header    X-Real-IP  $remote_addr;
         proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header    Host $http_host;
             proxy_redirect      off;
             proxy_pass          http://127.0.0.1:31539/;
         # kill cache
         add_header Last-Modified $date_gmt;
             add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
             if_modified_since off;
             expires off;
             etag off;
    }       

    location /backend {
            return 301 /app1/;
    }

What Should Happen Instead?

My expected result should be that after stop all containers will be shutdown, actually only the undeploy trough delete of deployments and service works.

Reproduction Steps

1. deploy a service with ingress implementation with traefik
2. launch microk8s stop
3. the kubernetes api are not more reacheble
4. the container api deployed in step 1 are still reacheble

inspection-report-20230515_094738.tar.gz

[neoaggelos]

Hi @muten84, thank you for reporting this issue

How are you stopping MicroK8s? microk8s stop should run the following and kill all running containers on your cluster, https://github.com/canonical/microk8s/blob/77dbbc9595b841af6052f8fe537a2eaa73fcb460/microk8s-resources/actions/common/utils.sh#L925-L926

Can you check two things on a system where this occurs:

microk8s stop

# look for running container processes
ps -fea | grep runc

Also, can you try running the following commands and observe their outputs?

sudo systemctl kill snap.microk8s.daemon-kubelite.service --signal=SIGKILL
sudo systemctl kill snap.microk8s.daemon-containerd.service --signal=SIGKILL

Thank you!

[SidMorad]

sudo snap stop microk8s --disable

then

sudo reboot

Not an ideal solution, but may help someone.

[rogerwegmanneit]

Same here, the containers are still running after "microk8s stop". Any news?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alrawas]

Please reopen. We're in version "MicroK8s v1.31.2 revision 7394" and the issue is still there.

[andrevcf]

Please reopen. Confirmed here with 1.31.2