search

kinvolk / lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk
https://kinvolk.io/lokomotive-kubernetes/
Apache License 2.0
321 stars 49 forks source link

Prometheus Operator component appears to require 2x 50GB volumes to work #351

Open schu opened 4 years ago

schu commented 4 years ago

i.e. the component doesn't work out of the box on many systems (for example when running on AWS with a smaller instance type like t3.micro). As far as I see there's no documentation for it, volume size can't be configured by the user and it's not immediately clear what is wrong; users have to find the event messages / forever pending volume claims:

kubectl get persistentvolumeclaims -n monitor
ing                                                                                                                
NAME                                                   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
data-alertmanager-prometheus-operator-alertmanager-0   Pending                                      openebs-default-sc   7m21s
data-prometheus-prometheus-operator-prometheus-0       Pending                                      openebs-default-sc   7m11s

CI uses i3.large as worker instance type, so it works there.

schu commented 4 years ago

I tried again with a modified lokoctl version with only 8GB per volume.

kubectl get events -n openebs |& grep Warning

12m         Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-547fc98c896sbdd     Error: cannot find volume "sockfile" to mount into container "cstor-istgt"
11m         Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-547fc98c896sbdd     Error: cannot find volume "sockfile" to mount into container "maya-volume-exporter"
11m         Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-547fc98c896sbdd     Error: cannot find volume "sockfile" to mount into container "cstor-volume-mgmt"
9m30s       Warning   FailedMount                    pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-6df7f4ff96wv2fn     Unable to attach or mount volumes: unmounted volumes=[tmp openebs-maya-operator-token-bnmlq sockfile conf], unattached volumes=[tmp openebs-maya-operator-token-bnmlq sockfile conf]: timed out waiting for the condition
5m31s       Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-758c55bf87zc47b     Error: cannot find volume "sockfile" to mount into container "cstor-istgt"
5m31s       Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-758c55bf87zc47b     Error: cannot find volume "sockfile" to mount into container "maya-volume-exporter"
5m31s       Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-758c55bf87zc47b     Error: cannot find volume "sockfile" to mount into container "cstor-volume-mgmt"
9m31s       Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-75d45d9bd-mxq4l     Error: cannot find volume "sockfile" to mount into container "cstor-istgt"
9m31s       Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-75d45d9bd-mxq4l     Error: cannot find volume "sockfile" to mount into container "maya-volume-exporter"
9m31s       Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-75d45d9bd-mxq4l     Error: cannot find volume "sockfile" to mount into container "cstor-volume-mgmt"
12m         Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-776db54c5-strmg     Error: cannot find volume "sockfile" to mount into container "cstor-istgt"
12m         Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-776db54c5-strmg     Error: cannot find volume "sockfile" to mount into container "maya-volume-exporter"
11m         Warning   Failed                         pod/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-target-776db54c5-strmg     Error: cannot find volume "sockfile" to mount into container "cstor-volume-mgmt"
13m         Warning   ClusterIPNotAllocated          service/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6                        Cluster IP 10.3.227.63 is not allocated; repairing
11m         Warning   FailedToUpdateEndpointSlices   service/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6                        Error updating Endpoint Slices for Service openebs/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6: Error updating pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-9fxjl EndpointSlice for Service openebs/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6: endpointslices.discovery.k8s.io "pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-9fxjl" not found
5m32s       Warning   FailedToUpdateEndpointSlices   service/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6                        Error updating Endpoint Slices for Service openebs/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6: Error updating pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-t8bb4 EndpointSlice for Service openebs/pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6: endpointslices.discovery.k8s.io "pvc-8099c0b9-bf5a-4615-8062-d4cae3be5de6-t8bb4" not found
10m         Warning   FailedMount                    pod/pvc-fb62ed50-04b5-4fac-b73f-34192e924ec8-target-57dcb6d744z99j5     Unable to attach or mount volumes: unmounted volumes=[conf tmp openebs-maya-operator-token-bnmlq sockfile], unattached volumes=[conf tmp openebs-maya-operator-token-bnmlq sockfile]: timed out waiting for the condition
10m         Warning   FailedMount                    pod/pvc-fb62ed50-04b5-4fac-b73f-34192e924ec8-target-756d84cd45qnbk6     Unable to attach or mount volumes: unmounted volumes=[sockfile conf tmp openebs-maya-operator-token-bnmlq], unattached volumes=[sockfile conf tmp openebs-maya-operator-token-bnmlq]: timed out waiting for the condition
11m         Warning   FailedCreatePodSandBox         pod/pvc-fb62ed50-04b5-4fac-b73f-34192e924ec8-target-764c66b595lnzc6     Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f720cc5f034df0c4c5bde0d62bb28c1c675c023e652559f87a6eac6adb2fe8b2" network for pod "pvc-fb62ed50-04b5-4fac-b73f-34192e924ec8-target-764c66b595lnzc6": networkPlugin cni failed to set up pod "pvc-fb62ed50-04b5-4fac-b73f-34192e924ec8-target-764c66b595lnzc6_openebs" network: failed to Statfs "/proc/14315/ns/net": no such file or directory
3m18s       Warning   FailedMount                    pod/pvc-fb62ed50-04b5-4fac-b73f-34192e924ec8-target-7fb69bf7dbgw5dw     Unable to attach or mount volumes: unmounted volumes=[tmp openebs-maya-operator-token-bnmlq sockfile conf], unattached volumes=[tmp openebs-maya-operator-token-bnmlq sockfile conf]: timed out waiting for the condition

OpenEBS doesn't seem to work.