Spark should not be exposed publicly

[c-w]

Currently Spark-Master and Spark-Zeppelin have publicly accessible services.

This means that anyone could submit jobs to the Fortis Spark cluster.

[c-w]

Currently deploying this change to validate.

[c-w]

Nope, that didn't work. Reverting.

[c-w]

Looks like turning the load-balancer into an internal one may solve the problem.

Implementation:

Add this:

  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

To the metadata block in the kubernetes service definition.

Confirmation that the pods don't crash:

[c-w]

Following up with some folks who know more about Kubernetes than I do to confirm if this is a good approach.

[c-w]

Confirmed with @jmspring that this is a good approach. Resolving.

[c-w]

commit 9a765922f6a27bebc0d924a023fa0ada3216cf0d
Author: Clemens Wolff <clewolff@microsoft.com>
Date:   Thu Dec 7 10:13:12 2017 -0500

    Hide external IPs of services

diff --git a/project-fortis-pipeline/ops/charts/cassandra/templates/svc.yaml b/project-fortis-pipeline/ops/charts/cassandra/templates/svc.yaml
index 7ff60e8..33f611c 100644
--- a/project-fortis-pipeline/ops/charts/cassandra/templates/svc.yaml
+++ b/project-fortis-pipeline/ops/charts/cassandra/templates/svc.yaml
@@ -25,6 +25,8 @@ metadata:
     chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
     release: "{{ .Release.Name }}"
     heritage: "{{ .Release.Service }}"
+  annotations:
+    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 spec:
   ports:
   - name: {{ template "fullname" . }}
diff --git a/project-fortis-pipeline/ops/charts/spark/templates/spark-master-service.yaml b/project-fortis-pipeline/ops/charts/spark/templates/spark-master-service.yaml
index 11c48a4..b36b245 100644
--- a/project-fortis-pipeline/ops/charts/spark/templates/spark-master-service.yaml
+++ b/project-fortis-pipeline/ops/charts/spark/templates/spark-master-service.yaml
@@ -7,6 +7,8 @@ metadata:
     release: {{.Release.Name | quote }}
     chart: "{{.Chart.Name}}-{{.Chart.Version}}"
     component: "{{.Values.Master.Component}}"
+  annotations:
+    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 spec:
   ports:
     - port: {{.Values.Master.ServicePort}}
diff --git a/project-fortis-pipeline/ops/charts/spark/templates/spark-zeppelin-deployment.yaml b/project-fortis-pipeline/ops/charts/spark/templates/spark-zeppelin-deployment.yaml
index 456d730..e5d0544 100644
--- a/project-fortis-pipeline/ops/charts/spark/templates/spark-zeppelin-deployment.yaml
+++ b/project-fortis-pipeline/ops/charts/spark/templates/spark-zeppelin-deployment.yaml
@@ -2,6 +2,8 @@ apiVersion: v1
 kind: Service
 metadata:
   name: zeppelin
+  annotations:
+    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 spec:
   ports:
     - port: {{.Values.Zeppelin.ServicePort}}