Closed johnhamelink closed 6 years ago
I think I've figured out what the issue is - I think that by default the /tmp/hostpath-provisioner
directory has the wrong ownership - changing the ownership to docker:docker
seems to fix things for me!
We can confirm there is an issue with persistent volume claims in minikube 0.24.1. We encountered the issue described after upgrading and attempting to deploy the concourse helm chart. This issue did not occur in minikube 0.23
Hitting this with kafka chart
"lstat" does not exist on Ubuntu 16.04.3 LTS. I used sudo ln -s /usr/bin/stat /usr/bin/lstat
but didn't help.
@johnhamelink I used --vm-driver=none
, but chmod -R 777 /tmp/hostpath-provisioner
didn't help either.
@southwolf I'm using minikube inside Virtualbox (I'm running arch and --vm-driver=none
was a headache I wasn't willing to work my way through just yet, lol).
To clarify, when I see the lstat
error, I'm running mkdir -p <directory>
then chown docker:docker /tmp/hostpath-provisioner
.
I'm hitting this as well. The interesting part is that I didn't hit it yesterday, but today it's hitting me. I'm using helm
to install the stable/postgresql
chart and that worked yesterday, but today I'm getting this error. I was able to verify earlier today that the volume existed in /tmp/hostpath-provisioner
but the sub paths were not being created.
I tore down my vm with minikube delete
and now nothing is being created in /tmp/hostpath-provisioner
. I then chmod 777 /tmp/hostpath*
reinstalled the chart and no go.
As a last ditch effort, I nuked my ~/.minikube
and still seeing the issue.
@grimmy Exactly the same here.
@grimmy Ran into this with postgres as well on 0.24.1
Any update on this bug?
Is there going to be a release anytime soon with this patch?
@grimmy Exactly the same here with Minikube 0.24.1
Error: lstat /tmp/hostpath-provisioner/pvc-6c84aa91-f04f-11e7-bf07-08002736d1ee: no such file or directory
I get this after "helm install stable/sonarqube" which also installs stable/postgresql
@southwolf when I follow your instructions the change doesn't persist over the restart. Is there anything else you did or something I am obviously missing?
@dyson same here . tried doing kubectl edit
, doesn't work either
Found the solution. @southwolf @dyson . Delete the storage-provisioner.yaml file from the Minikube VM and delete the pod associated with the file: kubectl delete pods/storage-provisioner -n kube-system
. And then insert the file into /etc/kubernetes/addons/. The storage-provisioner pod should restart by itself
@tarifrr I tried that and I'm still getting the error...
PersistentVolumeClaim is not bound: "fleetgrid-postgresql"
Error: lstat /tmp/hostpath-provisioner/pvc-afe84cbc-f308-11e7-b6ad-0800270b980e: no such file or directory
Error syncing pod
@trabur Could you tell me the steps you took?
Hitting a similar issue after trying the suggestion. No volumes created. My steps:
Remove the config file and kill the provisioner pod
minikube$ sudo rm /etc/kubernetes/addons/storage-provisioner.yaml
host$ kubectl delete pods/storage-provisioner -n kube-system
Ensure the pod has terminated
host$ kubectl get pods/storage-provisioner -n kube-system
Replace the provisioner config and install the chart
minikube$ sudo curl https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/storage-provisioner/storage-provisioner.yaml --output /etc/kubernetes/addons/storage-provisioner.yaml
host$ helm install stable/postgresql
Error reported
PersistentVolumeClaim is not bound: "sweet-goat-postgresql"
Unable to mount volumes for pod "sweet-goat-postgresql-764d89f465-f7fr2_default(4f0efe66-f460-11e7-b5f9-080027e117f4)": timeout expired waiting for volumes to attach/mount for pod "default"/"sweet-goat-postgresql-764d89f465-f7fr2". list of unattached/unmounted volumes=[data]
Error syncing pod
Check volumes and claims
host$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
sweet-goat-postgresql Pending 16m
host$ kubectl get pvc
No resources found.
@torstenek check whether the storage-provisioner pod is created first, if so then add in postgresql
Hello everyone!
I ran into the same issue. However I found that the problem was caused by configuration defaults. See this description.
Cause
I had created a PersistentVolumeClaim without a storageClassName
. Kubernetes then added a DefaultStorageClass named standard
to the claim.
The PersistentVolumes I had created did not have a storageClassName either. However those are not assigned a default. The storageClassName
in this case is equal to ""
or none.
As a result the claim could not find a matching PersistentVolume. Then Kubernetes created a new PersistentVolume with a hostPath similar to /tmp/hostpath-provisioner/pvc-name
. This directory did not exist hence the lstat error
.
Solution
Adding a storageClassName
to both the PersistentVolume
and PersistentVolume
spec solved the issue for me.
Hope this helps someone.
-Robert.
EDIT: The kubernetes.io page on persistent volumes also helped me to find the paths minikube allows for persistent volumes.
I followed the stackoverflow answer and it works
https://stackoverflow.com/questions/47849975/postgresql-via-helm-not-installing/48291156#48291156
This issue was fixed a while ago. When might we reasonably see a patch release of minikube? This is affecting a lot of people.
This has been released now in v0.25.0
@r2d4 awesome! Thank you!
@RobertDiebels thanks for your answer, i had to define a storageclass and use that in both my pvs and pvcs as well -- using minikube and minio. for anyone else that gets here, i also didnt have selectors in the pvc matching the labels in the pv correctly.
storage-provisioner is working in a fresh installed minikube.
but after days, all new created pvc is always pending. found storage-provisoner pod is missing.
Question: why the strorage-provisioner is started as pod only? no deployment or replicaset to maintain the replica of it.
@joshkendrick Happy to help 👍 . The hostpath-provisioner fix should sort out the issue though. I manually created host-path PV's before. Adding the proper storageclass for provisioning should allow PVC's to work without creating your own PV's now.
Does seem similar to https://github.com/kubernetes/minikube/issues/4634
BUG REPORT
Please provide the following details:
Environment:
Minikube version (use
minikube version
):minikube version: v0.24.1
cat ~/.minikube/machines/minikube/config.json | grep DriverName
):Virtualbox
cat ~/.minikube/machines/minikube/config.json | grep -i ISO
"Boot2DockerURL": "file:///home/john/.minikube/cache/iso/minikube-v0.23.6.iso"
minikube ssh cat /etc/VERSION
v0.23.6
Install tools:
Others:
What happened:
When attempting to install a pod resource, which has a VolumeMount with a subpath (like below):
The pod fails to bind to the volume, with the following error:
When subpath is not defined, this error does not happen.
What you expected to happen:
Creating a persisted volume claim with a subpath creates a directory which k8s can bind to.
How to reproduce it (as minimally and precisely as possible):
Output of
minikube logs
(if applicable):https://gist.github.com/johnhamelink/f8c3074d35ccb55f1203a4fa021b0cbb
Anything else do we need to know:
I was able to confirm that this issue didn't affect a macbook pro with the following versions:
I was able to get past this issue by manually creating the missing directory from iniside minikube by running
minikube ssh
.