search

kubernetes / website

Kubernetes website and documentation repo:
https://kubernetes.io
Creative Commons Attribution 4.0 International
4.47k stars 14.4k forks source link

Error: stat /tmp/hostpath-provisioner/default/data-mysql-0: no such file or directory #38188

Closed 0xfei closed 9 months ago

0xfei commented 1 year ago

I'm using Mac M1 and minkube, failed here

image

PV/PVC/SC seems work fine, service and configmap can upload, but stateful was failed.

➜  k8s kubectl get service
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP    5h50m
mysql        ClusterIP   None             <none>        3306/TCP   83m
mysql-read   ClusterIP   10.103.121.247   <none>        3306/TCP   83m
➜  k8s kubectl get configmaps 
NAME               DATA   AGE
kube-root-ca.crt   1      5h50m
mysql              2      84m
➜  k8s kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  5h50m
➜  k8s kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-mysql-0   Bound    pvc-0f8b20ef-6536-4379-9043-d2ade5e01395   5Gi        RWO            standard       6m4s
➜  k8s kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-0f8b20ef-6536-4379-9043-d2ade5e01395   5Gi        RWO            Delete           Bound    default/data-mysql-0   standard                6m6s

image

I run kubectl describe pod mysql-0 image

Error: stat /tmp/hostpath-provisioner/default/data-mysql-0: no such file or directory

k8s-ci-robot commented 1 year ago

@0xfei: This issue is currently awaiting triage.

SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
jihoon-seo commented 1 year ago

Link: https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/#statefulset

I tried to reproduce this but failed.

[env]

❯ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.24.8
Kustomize Version: v4.5.4
Server Version: v1.24.3
❯ minikube version
minikube version: v1.28.0
commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f

[Outputs]

❯ kubectl get pods -l app=mysql --watch
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          59s
mysql-1   2/2     Running   0          34s
mysql-2   2/2     Running   0          20s
❯ kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  3m51s
❯ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-mysql-0   Bound    pvc-2726201c-d1e8-4017-afa1-e908dd778600   10Gi       RWO            standard       4m49s
data-mysql-1   Bound    pvc-13c6fe1f-61d1-48a3-8da2-189d9d6f9d10   10Gi       RWO            standard       4m24s
data-mysql-2   Bound    pvc-a48f7ac3-d0b9-42e8-bda9-c516a3756ba5   10Gi       RWO            standard       4m10s
❯ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-13c6fe1f-61d1-48a3-8da2-189d9d6f9d10   10Gi       RWO            Delete           Bound    default/data-mysql-1   standard                5m1s
pvc-2726201c-d1e8-4017-afa1-e908dd778600   10Gi       RWO            Delete           Bound    default/data-mysql-0   standard                5m26s
pvc-a48f7ac3-d0b9-42e8-bda9-c516a3756ba5   10Gi       RWO            Delete           Bound    default/data-mysql-2   standard                4m47s
❯ minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
| ambassador                  | minikube | disabled     | 3rd party (Ambassador)         |
| auto-pause                  | minikube | disabled     | Google                         |
| cloud-spanner               | minikube | disabled     | Google                         |
| csi-hostpath-driver         | minikube | disabled     | Kubernetes                     |
| dashboard                   | minikube | disabled     | Kubernetes                     |
| default-storageclass        | minikube | enabled ✅   | Kubernetes                     |
| efk                         | minikube | disabled     | 3rd party (Elastic)            |
| freshpod                    | minikube | disabled     | Google                         |
| gcp-auth                    | minikube | disabled     | Google                         |
| gvisor                      | minikube | disabled     | Google                         |
| headlamp                    | minikube | disabled     | 3rd party (kinvolk.io)         |
| helm-tiller                 | minikube | disabled     | 3rd party (Helm)               |
| inaccel                     | minikube | disabled     | 3rd party (InAccel             |
|                             |          |              | [info@inaccel.com])            |
| ingress                     | minikube | disabled     | Kubernetes                     |
| ingress-dns                 | minikube | disabled     | Google                         |
| istio                       | minikube | disabled     | 3rd party (Istio)              |
| istio-provisioner           | minikube | disabled     | 3rd party (Istio)              |
| kong                        | minikube | disabled     | 3rd party (Kong HQ)            |
| kubevirt                    | minikube | disabled     | 3rd party (KubeVirt)           |
| logviewer                   | minikube | disabled     | 3rd party (unknown)            |
| metallb                     | minikube | disabled     | 3rd party (MetalLB)            |
| metrics-server              | minikube | disabled     | Kubernetes                     |
| nvidia-driver-installer     | minikube | disabled     | Google                         |
| nvidia-gpu-device-plugin    | minikube | disabled     | 3rd party (Nvidia)             |
| olm                         | minikube | disabled     | 3rd party (Operator Framework) |
| pod-security-policy         | minikube | disabled     | 3rd party (unknown)            |
| portainer                   | minikube | disabled     | 3rd party (Portainer.io)       |
| registry                    | minikube | disabled     | Google                         |
| registry-aliases            | minikube | disabled     | 3rd party (unknown)            |
| registry-creds              | minikube | disabled     | 3rd party (UPMC Enterprises)   |
| storage-provisioner         | minikube | enabled ✅   | Google                         |
| storage-provisioner-gluster | minikube | disabled     | 3rd party (Gluster)            |
| volumesnapshots             | minikube | disabled     | Kubernetes                     |
|-----------------------------|----------|--------------|--------------------------------|
0xfei commented 1 year ago 
Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  11s              default-scheduler  Successfully assigned **default/mysql-0 to demo-m03**
  Normal   Pulled     10s              kubelet            Container image "amd64/mysql:5.7" already present on machine
  Normal   Created    10s              kubelet            Created container init-mysql
  Normal   Started    10s              kubelet            Started container init-mysql
  Normal   Pulled     8s (x2 over 9s)  kubelet            Container image "gcr.io/google-samples/xtrabackup:1.0" already present on machine
  Warning  Failed     8s (x2 over 9s)  kubelet            **Error: stat /tmp/hostpath-provisioner/default/data-mysql-0**: no such file or directory

the cluster has three nodes:

➜  k8s kubectl get nodes 
NAME       STATUS   ROLES           AGE   VERSION
demo       Ready    control-plane   9h    v1.25.3
demo-m02   Ready    <none>          9h    v1.25.3
demo-m03   Ready    <none>          9h    v1.25.3

default/mysql-0 to demo-m03, demo is ready, but not demo-03:

on demo-03:

root@demo-m03:/# ls -al /tmp/hostpath-provisioner/
total 4
drwxr-xr-x 2 root root 4096 Nov 30 02:48 .
drwxrwxrwt 5 root root  140 Nov 30 08:58 ..

on demo:

root@demo:/# ls -al /tmp/hostpath-provisioner/default/
total 12
drwxr-xr-x 3 root root 4096 Nov 30 08:06 .
drwxr-xr-x 3 root root 4096 Nov 30 06:49 ..
drwxrwxrwx 2 root root 4096 Nov 30 08:06 data-mysql-0
0xfei commented 1 year ago

Warning FailedScheduling 10s default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

@jihoon-seo

kundan2707 commented 1 year ago

/kind support

0xfei commented 1 year ago

/kind support

@kundan2707 can you help me figure out what's going on, please.

I don't known why mysql-0 was not scheduled to demo(1st node of the cluster, and the control plane), and demo-03(3rd node of the cluster), THE CHOSEN ONE, did't create PVC/PV.

Kartik494 commented 1 year ago

@jihoon-seo i am not able to get my pvc bound when i am using local storage class , but as stated in the documentation that

You need to either have a dynamic PersistentVolume provisioner with a default StorageClass, or statically provision PersistentVolumes yourself to satisfy the PersistentVolumeClaims used here.

as you are using dynamic PV provisioner , could you also check on static provision pv ?

0xfei commented 1 year ago

@Kartik494 i am not able to get my pvc bound when i am using local storage class

The same with you, local storage cannot work either

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

kundan2707 commented 1 year ago

/remove-lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

sftim commented 11 months ago

I don't think we need to keep this issue alive. GitHub is not the right place for support requests @0xfei

If you're still looking for help, check Server Fault.

You can also post your question on the Kubernetes Slack or the Discuss Kubernetes forum.

/remove-lifecycle stale /lifecycle rotten

Gauravpadam commented 9 months ago

Closing in favour of https://github.com/kubernetes/website/issues/38188#issuecomment-1826382505

/close

k8s-ci-robot commented 9 months ago

@Gauravpadam: Closing this issue.

In response to [this](https://github.com/kubernetes/website/issues/38188#issuecomment-1893310781): >Closing in favour of https://github.com/kubernetes/website/issues/38188#issuecomment-1826382505 > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.