search

gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
875 stars 390 forks source link

Installation doesn't finish in k8s 1.14.1 #589

Open renich opened 5 years ago

renich commented 5 years ago

The issue is this: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#urgent-upgrade-notes

* kubectl
    - The deprecated --show-all flag to kubectl get has been removed (#69255, @Pingan2017)

Here's the deploy output:

# ./gk-deploy -n kube-system -gvy topology.json
Using Kubernetes CLI.

Checking status of namespace matching 'kube-system':
kube-system   Active   23m
Using namespace "kube-system".
Checking for pre-existing resources...
  GlusterFS pods ... 
Checking status of pods matching '--selector=glusterfs=pod':

Timed out waiting for pods matching '--selector=glusterfs=pod'.
not found.
  deploy-heketi pod ... 
Checking status of pods matching '--selector=deploy-heketi=pod':

Timed out waiting for pods matching '--selector=deploy-heketi=pod'.
not found.
  heketi pod ... 
Checking status of pods matching '--selector=heketi=pod':

Timed out waiting for pods matching '--selector=heketi=pod'.
not found.
  gluster-s3 pod ... 
Checking status of pods matching '--selector=glusterfs=s3-pod':

Timed out waiting for pods matching '--selector=glusterfs=s3-pod'.
not found.
Creating initial resources ... /usr/bin/kubectl -n kube-system create -f /root/src/gluster-kubernetes/deploy/kube-templates/heketi-service-account.yaml 2>&1
serviceaccount/heketi-service-account created
/usr/bin/kubectl -n kube-system create clusterrolebinding heketi-sa-view --clusterrole=edit --serviceaccount=kube-system:heketi-service-account 2>&1
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view created
/usr/bin/kubectl -n kube-system label --overwrite clusterrolebinding heketi-sa-view glusterfs=heketi-sa-view heketi=sa-view
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view labeled
OK
Marking 'glusterfs0.test.mia.cloudsigma.com' as a GlusterFS node.
/usr/bin/kubectl -n kube-system label nodes glusterfs0.test.mia.cloudsigma.com storagenode=glusterfs --overwrite 2>&1
node/glusterfs0.test.mia.cloudsigma.com labeled
Marking 'glusterfs1.test.mia.cloudsigma.com' as a GlusterFS node.
/usr/bin/kubectl -n kube-system label nodes glusterfs1.test.mia.cloudsigma.com storagenode=glusterfs --overwrite 2>&1
node/glusterfs1.test.mia.cloudsigma.com labeled
Marking 'glusterfs2.test.mia.cloudsigma.com' as a GlusterFS node.
/usr/bin/kubectl -n kube-system label nodes glusterfs2.test.mia.cloudsigma.com storagenode=glusterfs --overwrite 2>&1
node/glusterfs2.test.mia.cloudsigma.com labeled
Deploying GlusterFS pods.
sed -e 's/storagenode\: glusterfs/storagenode\: 'glusterfs'/g' /root/src/gluster-kubernetes/deploy/kube-templates/glusterfs-daemonset.yaml | /usr/bin/kubectl -n kube-system create -f - 2>&1
daemonset.extensions/glusterfs created
Waiting for GlusterFS pods to start ... 
Checking status of pods matching '--selector=glusterfs=pod':
glusterfs-7xtct   1/1   Running   0     67s
glusterfs-zcpfn   1/1   Running   0     67s
glusterfs-zg6c7   1/1   Running   0     67s
OK
/usr/bin/kubectl -n kube-system create secret generic heketi-config-secret --from-file=private_key=/dev/null --from-file=./heketi.json --from-file=topology.json=topology.json
secret/heketi-config-secret created
/usr/bin/kubectl -n kube-system label --overwrite secret heketi-config-secret glusterfs=heketi-config-secret heketi=config-secret
secret/heketi-config-secret labeled
sed -e 's/\${HEKETI_EXECUTOR}/kubernetes/' -e 's#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#' -e 's/\${HEKETI_ADMIN_KEY}//' -e 's/\${HEKETI_USER_KEY}//' /root/src/gluster-kubernetes/deploy/kube-templates/deploy-heketi-deployment.yaml | /usr/bin/kubectl -n kube-system create -f - 2>&1
service/deploy-heketi created
deployment.extensions/deploy-heketi created
Waiting for deploy-heketi pod to start ... 
Checking status of pods matching '--selector=deploy-heketi=pod':
deploy-heketi-865f55765-6tddb   1/1   Running   0     8s
OK
Determining heketi service URL ... OK
/usr/bin/kubectl -n kube-system exec -i deploy-heketi-865f55765-6tddb -- heketi-cli -s http://localhost:8080 --user admin --secret '' topology load --json=/etc/heketi/topology.json 2>&1
Creating cluster ... ID: fd555fae60338786423a8ac4495b3c7d
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node glusterfs0.test.mia.cloudsigma.com ... ID: c48a67586d94d62700571bcacc0b5c25
Adding device /dev/vdb ... OK
Creating node glusterfs1.test.mia.cloudsigma.com ... ID: 5d171308d6b10522fc555d35f6e9e2a6
Adding device /dev/vdb ... OK
Creating node glusterfs2.test.mia.cloudsigma.com ... ID: 1ebdea31a7d5b3d2f3334d763e8713f8
Adding device /dev/vdb ... OK
heketi topology loaded.
/usr/bin/kubectl -n kube-system exec -i deploy-heketi-865f55765-6tddb -- heketi-cli -s http://localhost:8080 --user admin --secret '' setup-openshift-heketi-storage --listfile=/tmp/heketi-storage.json  2>&1
Saving /tmp/heketi-storage.json
/usr/bin/kubectl -n kube-system exec -i deploy-heketi-865f55765-6tddb -- cat /tmp/heketi-storage.json | /usr/bin/kubectl -n kube-system create -f - 2>&1
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

Checking status of pods matching '--selector=job-name=heketi-storage-copy-job':
heketi-storage-copy-job-ckf78   0/1   Completed   0     4s
/usr/bin/kubectl -n kube-system label --overwrite svc heketi-storage-endpoints glusterfs=heketi-storage-endpoints heketi=storage-endpoints
service/heketi-storage-endpoints labeled
/usr/bin/kubectl -n kube-system delete all,service,jobs,deployment,secret --selector="deploy-heketi" 2>&1
pod "deploy-heketi-865f55765-6tddb" deleted
service "deploy-heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-865f55765" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted
sed -e 's/\${HEKETI_EXECUTOR}/kubernetes/' -e 's#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#' -e 's/\${HEKETI_ADMIN_KEY}//' -e 's/\${HEKETI_USER_KEY}//' /root/src/gluster-kubernetes/deploy/kube-templates/heketi-deployment.yaml | /usr/bin/kubectl -n kube-system create -f - 2>&1
service/heketi created
deployment.extensions/heketi created
Waiting for heketi pod to start ... 
Checking status of pods matching '--selector=heketi=pod':
heketi-85dbbbb55-s56cr   1/1   Running   0     10s
OK
Determining heketi service URL ... Error: unknown flag: --show-all

Examples:
  # List all pods in ps output format.
  kubectl get pods

  # List all pods in ps output format with more information (such as node name).
  kubectl get pods -o wide

  # List a single replication controller with specified NAME in ps output format.
  kubectl get replicationcontroller web

  # List deployments in JSON output format, in the "v1" version of the "apps" API group:
  kubectl get deployments.v1.apps -o json

  # List a single pod in JSON output format.
  kubectl get -o json pod web-pod-13je7

  # List a pod identified by type and name specified in "pod.yaml" in JSON output format.
  kubectl get -f pod.yaml -o json

  # List resources from a directory with kustomization.yaml - e.g. dir/kustomization.yaml.
  kubectl get -k dir/

  # Return only the phase value of the specified pod.
  kubectl get -o template pod/web-pod-13je7 --template={{.status.phase}}

  # List resource information in custom columns.
  kubectl get pod test-pod -o custom-columns=CONTAINER:.spec.containers[0].name,IMAGE:.spec.containers[0].image

  # List all replication controllers and services together in ps output format.
  kubectl get rc,services

  # List one or more resources by their type and names.
  kubectl get rc/web service/frontend pods/web-pod-13je7

Options:
  -A, --all-namespaces=false: If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace.
      --allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.
      --chunk-size=500: Return large lists in chunks rather than all at once. Pass 0 to disable. This flag is beta and may change in the future.
      --field-selector='': Selector (field query) to filter on, supports '=', '==', and '!='.(e.g. --field-selector key1=value1,key2=value2). The server only supports a limited number of field queries per type.
  -f, --filename=[]: Filename, directory, or URL to files identifying the resource to get from a server.
      --ignore-not-found=false: If the requested object does not exist the command will return exit code 0.
  -k, --kustomize='': Process the kustomization directory. This flag can't be used together with -f or -R.
  -L, --label-columns=[]: Accepts a comma separated list of labels that are going to be presented as columns. Names are case-sensitive. You can also use multiple flag options like -L label1 -L label2...
      --no-headers=false: When using the default or custom-column output format, don't print headers (default print headers).
  -o, --output='': Output format. One of: json|yaml|wide|name|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=... See custom columns [http://kubernetes.io/docs/user-guide/kubectl-overview/#custom-columns], golang template [http://golang.org/pkg/text/template/#pkg-overview] and jsonpath template [http://kubernetes.io/docs/user-guide/jsonpath].
      --raw='': Raw URI to request from the server.  Uses the transport specified by the kubeconfig file.
  -R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.
  -l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2)
      --server-print=true: If true, have the server return the appropriate table output. Supports extension APIs and CRDs.
      --show-kind=false: If present, list the resource type for the requested object(s).
      --show-labels=false: When printing, show all labels as the last column (default hide labels column)
      --sort-by='': If non-empty, sort list types using this field specification.  The field specification is expressed as a JSONPath expression (e.g. '{.metadata.name}'). The field in the API resource specified by this JSONPath expression must be an integer or a string.
      --template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].
  -w, --watch=false: After listing/getting the requested object, watch for changes. Uninitialized objects are excluded if no object name is provided.
      --watch-only=false: Watch for changes to the requested object(s), without listing/getting first.

Usage:
  kubectl get [(-o|--output=)json|yaml|wide|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...] (TYPE[.VERSION][.GROUP] [NAME | -l label] | TYPE[.VERSION][.GROUP]/NAME ...) [flags] [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).

unknown flag: --show-all
Failed to communicate with heketi service.
renich commented 5 years ago

Removing the --show-all, effectively allows me to finish the installation:

# ./gk-deploy -n kube-system -gvy topology.json
Using Kubernetes CLI.

Checking status of namespace matching 'kube-system':
kube-system   Active   50m
Using namespace "kube-system".
Checking for pre-existing resources...
  GlusterFS pods ... 
Checking status of pods matching '--selector=glusterfs=pod':

Timed out waiting for pods matching '--selector=glusterfs=pod'.
not found.
  deploy-heketi pod ... 
Checking status of pods matching '--selector=deploy-heketi=pod':

Timed out waiting for pods matching '--selector=deploy-heketi=pod'.
not found.
  heketi pod ... 
Checking status of pods matching '--selector=heketi=pod':

Timed out waiting for pods matching '--selector=heketi=pod'.
not found.
  gluster-s3 pod ... 
Checking status of pods matching '--selector=glusterfs=s3-pod':

Timed out waiting for pods matching '--selector=glusterfs=s3-pod'.
not found.
Creating initial resources ... /usr/bin/kubectl -n kube-system create -f /root/src/gluster-kubernetes/deploy/kube-templates/heketi-service-account.yaml 2>&1
serviceaccount/heketi-service-account created
/usr/bin/kubectl -n kube-system create clusterrolebinding heketi-sa-view --clusterrole=edit --serviceaccount=kube-system:heketi-service-account 2>&1
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view created
/usr/bin/kubectl -n kube-system label --overwrite clusterrolebinding heketi-sa-view glusterfs=heketi-sa-view heketi=sa-view
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view labeled
OK
Marking 'glusterfs0.test.mia.cloudsigma.com' as a GlusterFS node.
/usr/bin/kubectl -n kube-system label nodes glusterfs0.test.mia.cloudsigma.com storagenode=glusterfs --overwrite 2>&1
node/glusterfs0.test.mia.cloudsigma.com labeled
Marking 'glusterfs1.test.mia.cloudsigma.com' as a GlusterFS node.
/usr/bin/kubectl -n kube-system label nodes glusterfs1.test.mia.cloudsigma.com storagenode=glusterfs --overwrite 2>&1
node/glusterfs1.test.mia.cloudsigma.com labeled
Marking 'glusterfs2.test.mia.cloudsigma.com' as a GlusterFS node.
/usr/bin/kubectl -n kube-system label nodes glusterfs2.test.mia.cloudsigma.com storagenode=glusterfs --overwrite 2>&1
node/glusterfs2.test.mia.cloudsigma.com labeled
Deploying GlusterFS pods.
sed -e 's/storagenode\: glusterfs/storagenode\: 'glusterfs'/g' /root/src/gluster-kubernetes/deploy/kube-templates/glusterfs-daemonset.yaml | /usr/bin/kubectl -n kube-system create -f - 2>&1
daemonset.extensions/glusterfs created
Waiting for GlusterFS pods to start ... 
Checking status of pods matching '--selector=glusterfs=pod':
glusterfs-lbkxk   1/1   Running   0     59s
glusterfs-m424d   1/1   Running   0     59s
glusterfs-s5mcq   1/1   Running   0     59s
OK
/usr/bin/kubectl -n kube-system create secret generic heketi-config-secret --from-file=private_key=/dev/null --from-file=./heketi.json --from-file=topology.json=topology.json
secret/heketi-config-secret created
/usr/bin/kubectl -n kube-system label --overwrite secret heketi-config-secret glusterfs=heketi-config-secret heketi=config-secret
secret/heketi-config-secret labeled
sed -e 's/\${HEKETI_EXECUTOR}/kubernetes/' -e 's#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#' -e 's/\${HEKETI_ADMIN_KEY}//' -e 's/\${HEKETI_USER_KEY}//' /root/src/gluster-kubernetes/deploy/kube-templates/deploy-heketi-deployment.yaml | /usr/bin/kubectl -n kube-system create -f - 2>&1
service/deploy-heketi created
deployment.extensions/deploy-heketi created
Waiting for deploy-heketi pod to start ... 
Checking status of pods matching '--selector=deploy-heketi=pod':
deploy-heketi-865f55765-7m9sk   1/1   Running   0     10s
OK
Determining heketi service URL ... OK
/usr/bin/kubectl -n kube-system exec -i deploy-heketi-865f55765-7m9sk -- heketi-cli -s http://localhost:8080 --user admin --secret '' topology load --json=/etc/heketi/topology.json 2>&1
Creating cluster ... ID: d82ac8a721d080bbaf762132910d7b9f
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node glusterfs0.test.mia.cloudsigma.com ... ID: 47fb2cb2f4738b64095bcaca3d3fba55
Adding device /dev/vdb ... OK
Creating node glusterfs1.test.mia.cloudsigma.com ... ID: 8dc94b24f65e5134942bedd2cffc0679
Adding device /dev/vdb ... OK
Creating node glusterfs2.test.mia.cloudsigma.com ... ID: d150b9083d7ce55073ef2da26fac0f9a
Adding device /dev/vdb ... OK
heketi topology loaded.
/usr/bin/kubectl -n kube-system exec -i deploy-heketi-865f55765-7m9sk -- heketi-cli -s http://localhost:8080 --user admin --secret '' setup-openshift-heketi-storage --listfile=/tmp/heketi-storage.json  2>&1
Saving /tmp/heketi-storage.json
/usr/bin/kubectl -n kube-system exec -i deploy-heketi-865f55765-7m9sk -- cat /tmp/heketi-storage.json | /usr/bin/kubectl -n kube-system create -f - 2>&1
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

Checking status of pods matching '--selector=job-name=heketi-storage-copy-job':
heketi-storage-copy-job-29bv7   0/1   Completed   0     4s
/usr/bin/kubectl -n kube-system label --overwrite svc heketi-storage-endpoints glusterfs=heketi-storage-endpoints heketi=storage-endpoints
service/heketi-storage-endpoints labeled
/usr/bin/kubectl -n kube-system delete all,service,jobs,deployment,secret --selector="deploy-heketi" 2>&1
pod "deploy-heketi-865f55765-7m9sk" deleted
service "deploy-heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-865f55765" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted
sed -e 's/\${HEKETI_EXECUTOR}/kubernetes/' -e 's#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#' -e 's/\${HEKETI_ADMIN_KEY}//' -e 's/\${HEKETI_USER_KEY}//' /root/src/gluster-kubernetes/deploy/kube-templates/heketi-deployment.yaml | /usr/bin/kubectl -n kube-system create -f - 2>&1
service/heketi created
deployment.extensions/heketi created
Waiting for heketi pod to start ... 
Checking status of pods matching '--selector=heketi=pod':
heketi-85dbbbb55-l787t   1/1   Running   0     13s
OK
Determining heketi service URL ... OK

heketi is now running and accessible via http://10.244.4.8:8080 . To run
administrative commands you can install 'heketi-cli' and use it as follows:

  # heketi-cli -s http://10.244.4.8:8080 --user admin --secret '<ADMIN_KEY>' cluster list

You can find it at https://github.com/heketi/heketi/releases . Alternatively,
use it from within the heketi pod:

  # /usr/bin/kubectl -n kube-system exec -i heketi-85dbbbb55-l787t -- heketi-cli -s http://localhost:8080 --user admin --secret '<ADMIN_KEY>' cluster list

For dynamic provisioning, create a StorageClass similar to this:

---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.244.4.8:8080"

Deployment complete!

Here's the diff:

diff --git a/deploy/gk-deploy b/deploy/gk-deploy
index 8cf5b40..29f84e7 100755
--- a/deploy/gk-deploy
+++ b/deploy/gk-deploy
@@ -916,7 +916,7 @@ while [[ "x${heketi_service}" == "x" ]] || [[ "${heketi_service}" == "<none>" ]]
   heketi_service=$(${CLI} describe svc/heketi | grep "Endpoints:" | awk '{print $2}')
 done

-heketi_pod=$(${CLI} get pod --no-headers --show-all --selector="heketi" | awk '{print $1}')
+heketi_pod=$(${CLI} get pod --no-headers --selector="heketi" | awk '{print $1}')

 if [[ "${CLI}" == *oc\ * ]]; then
   heketi_service=$(${CLI} describe routes/heketi | grep "Requested Host:" | awk '{print $3}')
baddlan commented 5 years ago

Related: https://github.com/gluster/gluster-kubernetes/issues/582