search

IBM / ibm-storage-odf-console

ibm-storage-odf-console provides IBM storage specific console page, which will be loaded by ODF console when end users access IBM storage. It's specially designed for displaying IBM specific storage attributes to customer. Current scope includes IBM flashsystem only.
Apache License 2.0
3 stars 7 forks source link

Spectrum scale core and gui pods stuck in init. Missing secret. #39

Closed MonicaLemay closed 3 years ago

MonicaLemay commented 3 years ago

All of the scale core pods and gui pods are stuck in init. There are no obvious errors in the gui pod. The scale core pod mention a missing secret. I assume it is the operator that creates the secret.

[root@cluster2-inf ocs-ci]# oc get pod -n ibm-spectrum-scale  | grep -v csi
NAME                                               READY   STATUS     RESTARTS   AGE
ibm-spectrum-scale-7g8wl                           0/1     Init:0/2   0          13d
ibm-spectrum-scale-core-operator-d5559db7d-d8rlj   1/1     Running    6          13d
ibm-spectrum-scale-lbrt9                           0/1     Init:0/2   0          13d
ibm-spectrum-scale-mp5pc                           0/1     Init:0/2   0          13d
ibm-spectrum-scale-znn28                           0/1     Init:0/2   0          13d
scale-gui-0                                        0/9     Init:0/1   3          13d
scale-pmcollector-0                                2/2     Running    0          13d
scale-pmcollector-1                                2/2     Running    0          13d

oc describe says that a secret doesn't exist. But our instructions didn't say to create a secret. I think the operator is doing that b/c the name of the secret that it is looking for is not something I would have chosen: scale-core-token-87bkd

Events:
  Type     Reason       Age                      From     Message
  ----     ------       ----                     ----     -------
  Warning  FailedMount  14m (x4655 over 6d8h)    kubelet  (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[scale-core-token-87bkd], unattached volumes=[asm-generic modules gpfs0 scale-core-token-87bkd gpfssystemlog core-config scale-pmsensors-volume asm linux-include gpfslog gui-config kernels ssh-keys mmfs overlay]: timed out waiting for the condition
  Warning  FailedMount  3m48s (x3890 over 6d8h)  kubelet  MountVolume.SetUp failed for volume "scale-core-token-87bkd" : secret "scale-core-token-87bkd" not found
[root@cluster2-inf ~]#

These are my secrets.

[root@cluster2-inf ocs-ci]# oc get secret -n ibm-spectrum-scale   | grep scale-core
scale-core-dockercfg-cdzbn                           kubernetes.io/dockercfg               1      13d
scale-core-token-cmp9j                               kubernetes.io/service-account-token   4      13d
scale-core-token-g7xv5                               kubernetes.io/service-account-token   4      13d
[root@cluster2-inf ocs-ci]#

CSI pods are all running with no problems.

[root@cluster2-inf ocs-ci]# oc get pod -n ibm-spectrum-scale  | grep csi
ibm-spectrum-scale-csi-attacher-0                  1/1     Running    3          13d
ibm-spectrum-scale-csi-bghw4                       2/2     Running    2          28d
ibm-spectrum-scale-csi-j4tmh                       2/2     Running    2          28d
ibm-spectrum-scale-csi-j9qvv                       2/2     Running    8          28d
ibm-spectrum-scale-csi-operator-659586f554-6dnsh   1/1     Running    14         13d
ibm-spectrum-scale-csi-provisioner-0               1/1     Running    3          13d
ibm-spectrum-scale-csi-snapshotter-0               1/1     Running    3          13d
ibm-spectrum-scale-csi-xxg68                       2/2     Running    2          28d
[root@cluster2-inf ocs-ci]#

On a side note, should CSI start before Spectrum Scale? Maybe that is OK but not sure. In previous projects, CNSA (cloud native Spectrum Scale), CSI had a dependency on Spectrum scale . Anyway that is not what I am reporting here.

What is at issue here is that the Spectrum Scale pods are stuck in Init.

MonicaLemay commented 3 years ago

Closing issues this was left over on the clsuter from previous CNSA testing and not related to ODF