Heketi failed to create volume: internal error - server connection terminated

obergeru commented 5 years ago

When i try to setup a dynamic provisioned volume with heketi-gluster storage class, the volume does not get created but stay in "pending state"

Name:          gluster1
Namespace:     default
StorageClass:  gluster-heketi
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: gluster-heketi
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  ProvisioningFailed  35s (x2 over 110s)  persistentvolume-controller  Failed to provision volume with StorageClass "gluster-heketi": failed to create volume: failed to create volume: <HEAD><TITLE>internal error - server connection terminated</TITLE></HEAD>
<BODY BGCOLOR="white" FGCOLOR="black"><H1>internal error - server connection terminated</H1><HR>
<FONT FACE="Helvetica,Arial"><B>
Description: internal error - server connection terminated</B></FONT>
<HR>
<!-- default "internal error - server connection terminated" response (502) -->
</BODY>

Mounted By:  <none>

[kubernetes@ald-kub-01 bin]$ kubectl describe storageclass
Name:                  gluster-heketi
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           kubernetes.io/glusterfs
Parameters:            restauthenabled=false,resturl=http://10.233.79.76:8080
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

[kubernetes@ald-kub-01 bin]$ export HEKETI_CLI_SERVER=$(kubectl -n glusterfs describe svc heketi |awk '/Endpoints/ {print "http://"$2}')

[kubernetes@ald-kub-01 bin]$ echo $HEKETI_CLI_SERVER
http://10.233.79.76:8080

When using the heketi cli on master server for static volume it seems to work;

[kubernetes@ald-kub-01 bin]$ ./heketi-cli volume create --size=1 --persistent-volume --persistent-volume-endpoint=heketi-storage-endpoints |kubectl create -f -
persistentvolume/glusterfs-733ea120 created
[kubernetes@ald-kub-01 bin]$ kubectl describe pv
Name:            glusterfs-733ea120

Labels:          <none>
Annotations:     <none>
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:
Status:          Available
Claim:
Reclaim Policy:  Retain
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:
Source:
    Type:                Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:       heketi-storage-endpoints
    EndpointsNamespace:  <nil>
    Path:                vol_733ea120c5a2b86e8eb720180ccfc4da
    ReadOnly:            false
Events:                  <none>

obergeru commented 5 years ago

After some more inspection I suspect that the problem is proxy related due to 10.xxx. is not in the no_proxy list of kubernetes manager

maurya-m commented 5 years ago

hi @obergeru , i am also seeing similar issue provisioning volume , what are your logs reporting in /var/log/glusterfs?

i am seeing this error where its Samba is not installed!

E [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0xe2c9a) [0x7f3d5ac5fc9a] -->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0xe26c3) [0x7f3d5ac5f6c3] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f3d5fdcb0f5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_4eadd2be53ce8fd0a339bcc725cf3932 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd

@jarrpa do we need to have Samba installed for the dynamic provisioning? using latest gluster image here.

poligraph commented 4 years ago

I have got almost the same error as @obergeru got. I was able to create pv manually with heketi-cli, but when I try to create pvc I see following:

kubectl describe pvc
Name:          gluster1
Namespace:     default
StorageClass:
Status:        Pending
Volume:
Labels:        <none>
Annotations:   storageClassName: gluster-heketi
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  3m49s (x62 over 18m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

phlogistonjohn commented 4 years ago

@maurya-m Samba is not required nor recommended for provisioning gluster volumes with heketi. That error seems to be a problem running a hook script in gluster. Is this a standalone gluster setup you built or is it a containerized gluster where you see this logging?

@poligraph This does not appear to be the same error. Your error has the text "no storage class is set" which indicates your PVC was not set up correctly for dynamic provisioning. Please review your PVC yaml your cluster needs either a default storage class or the storage class to be specificd in the pvc.

gluster / gluster-kubernetes

Heketi failed to create volume: internal error - server connection terminated #575