search

openshift / origin-metrics

79 stars 113 forks source link

starting metrics in USE_PERSISTENT_STORAGE=true mode will fail if pvc with specific name metrics-cassandra-1 is not created in advance #119

Closed ekuric closed 8 years ago

ekuric commented 8 years ago

when trying to start metrics with persistant storage using USE_PERSISTENT_STORAGE=true eg by executing

# oc process -f metrics.yaml -v HAWKULAR_METRICS_HOSTNAME=hawkular_hostname,USE_PERSISTENT_STORAGE=true | oc create -n openshift-infra -f -

it will fail to start cassandra pod if previously PVC with specific name is not created on system requested pvc name has to be : metrics-cassandra-1

if this pvc with that specific pvc name is present before executing above command , then Cassandra pod will start fine, otherwise will hang and fail with [1] Also, it seems it is not possible to specify PVC name when starting metrics in persistent storage mode

[1]

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

40s 40s 1 {default-scheduler } Normal Scheduled Successfully assigned hawkular-cassandra-1-vz5en to ip-x.x.x.us-west-2.compute.internal 40s 3s 4 {kubelet ip-x.x.x.x.us-west-2.compute.internal} Warning FailedMount Unable to mount volumes for pod "hawkular-cassandra-1-vz5en_openshift-infra(117e3dc9-1363-11e6-ba84-02344e4803fd)": failed to instantiate volume builder for cassandra-data: The claim metrics-cassandra-1 is not yet bound to a volume 40s 3s 4 {kubelet ip-x.x.x.x.us-west-2.compute.internal} Warning FailedSync Error syncing pod, skipping: failed to instantiate volume builder for cassandra-data: The claim metrics-cassandra-1 is not yet bound to a volume`

ekuric commented 8 years ago

oc get pv; oc get pvc; oc get pods 

 oc get pods; oc get pv ; oc get pvc
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-h3ebf   1/1       Running     0          2h
hawkular-metrics-tu0z7       1/1       Running     0          2h
heapster-pyg6c               1/1       Running     0          2h
metrics-deployer-bjol0       0/1       Completed   0          2h
NAME       CAPACITY   ACCESSMODES   STATUS    CLAIM                                 REASON    AGE
pvmetric   30Gi       RWO           Bound     openshift-infra/metrics-cassandra-1             3h
NAME                  STATUS    VOLUME     CAPACITY   ACCESSMODES   AGE
metrics-cassandra-1   Bound     pvmetric   30Gi       RWO           3h
mwringe commented 8 years ago

The installer should create the PVC for you if it does not already exist. What version of OpenShift and the metrics components are you using?

mwringe commented 8 years ago

Also note: "failed to instantiate volume builder for cassandra-data: The claim metrics-cassandra-1 is not yet bound to a volume"

Are you sure you have a PV configured on your system which is of a sufficient size? There needs to be a PV available so that the PVC can use it.

ekuric commented 8 years ago

@mwringe I created pv "pvmetric" , prior running below oc process -f metrics.yaml -v HAWKULAR_METRICS_HOSTNAME=hawkular_hostname,USE_PERSISTENT_STORAGE=true | oc create -n openshift-infra -f -

I used https://github.com/openshift/origin-metrics and Openshift v3.2 packages

ekuric commented 8 years ago

@mwringe @jcantrill I am not getting this issue again. I did not remember I checked 

oc get pvc --all-namespaces

I suspect, there were pvc with specific name and it was tried again to bind to it. Cannot reproduce it again, ... vote to close this one?

jcantrill commented 8 years ago

@mwringe I am fine closing unless you have objections