apache-spark-on-k8s / kubernetes-HDFS

Repository holding configuration files for running an HDFS cluster in Kubernetes
Apache License 2.0
397 stars 185 forks source link

Can't get a working deployment - networking issues #73

Open jeffsaremi opened 5 years ago

jeffsaremi commented 5 years ago

Installed using the command below and got the pasted results


$ helm install --name hdtest --namespace hdns charts/hdfs-k8s
NAME:   hdtest
LAST DEPLOYED: Wed Jul 31 16:05:09 2019
NAMESPACE: hdns
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                          DATA  AGE
hdtest-hdfs-config            2     0s
hdtest-hdfs-datanode-scripts  1     0s
hdtest-hdfs-namenode-scripts  3     0s

==> v1/Pod(related)
NAME                                 READY  STATUS             RESTARTS  AGE
hdtest-hdfs-client-85cdf9488f-tjqpk  0/1    ContainerCreating  0         0s
hdtest-hdfs-datanode-79fjr           0/1    ContainerCreating  0         0s
hdtest-hdfs-journalnode-0            0/1    Pending            0         0s
hdtest-hdfs-namenode-0               0/1    Pending            0         0s
hdtest-zookeeper-0                   0/1    Pending            0         0s

==> v1/Service
NAME                       TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)                     AGE
hdtest-hdfs-journalnode    ClusterIP  None          <none>       8485/TCP,8480/TCP           0s
hdtest-hdfs-namenode       ClusterIP  None          <none>       8020/TCP,50070/TCP          0s
hdtest-zookeeper           ClusterIP  10.102.44.35  <none>       2181/TCP                    0s
hdtest-zookeeper-headless  ClusterIP  None          <none>       2181/TCP,3888/TCP,2888/TCP  0s

==> v1beta1/DaemonSet
NAME                  DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
hdtest-hdfs-datanode  1        1        0      1           0          <none>         0s

==> v1beta1/Deployment
NAME                READY  UP-TO-DATE  AVAILABLE  AGE
hdtest-hdfs-client  0/1    1           0          0s

==> v1beta1/PodDisruptionBudget
NAME                     MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
hdtest-hdfs-journalnode  2              N/A              0                    0s
hdtest-hdfs-namenode     1              N/A              0                    0s
hdtest-zookeeper         N/A            1                0                    0s

==> v1beta1/StatefulSet
NAME                     READY  AGE
hdtest-hdfs-journalnode  0/3    0s
hdtest-hdfs-namenode     0/2    0s
hdtest-zookeeper         0/3    0s

The list of pods I get is a lot shorter than the one listed in Readme


$ kubectl get pods -n hdns -o wide
NAME                                  READY   STATUS    RESTARTS   AGE    IP              NODE          NOMINATED NODE   READINESS GATES
hdtest-hdfs-client-85cdf9488f-tjqpk   1/1     Running   0          115s   10.244.0.22     js-u16exp-1   <none>           <none>
hdtest-hdfs-datanode-79fjr            0/1     Running   0          115s   10.160.103.55   js-u16exp-1   <none>           <none>
hdtest-hdfs-journalnode-0             0/1     Pending   0          115s   <none>          <none>        <none>           <none>
hdtest-hdfs-namenode-0                0/1     Pending   0          115s   <none>          <none>        <none>           <none>
hdtest-zookeeper-0                    0/1     Pending   0          115s   <none>          <none>        <none>           <none>

and if I do a dfsamin report I get this:


$ kubectl exec $_CLIENT -n hdns -- hdfs dfsadmin -report
19/07/31 23:11:32 WARN hdfs.DFSUtil: Namenode for hdfs-k8s remains unresolved for ID nn0.  Check your hdfs-site.xml file to ensure namenodes are configured properly.
19/07/31 23:11:37 WARN hdfs.DFSUtil: Namenode for hdfs-k8s remains unresolved for ID nn1.  Check your hdfs-site.xml file to ensure namenodes are configured properly.
report: java.net.UnknownHostException: hdtest-hdfs-namenode-1.hdtest-hdfs-namenode.hdns.svc.cluster.local
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning]
command terminated with exit code 255