green end-end system tests

[jldec]

For developers on the PFS team to get value out of Ci we need end-to-end tests which exercise as much of the whole system as possible.

[freynca]

• System tests currently use an outdated sample function (https://github.com/pivotal-cf/pfs-ci/tree/master/integration/sample_functions/java/uppercase)
• Will switch to using the Greeter sample from sk8s (https://github.com/markfisher/sk8s/tree/master/samples/java/greeter)

[freynca]

Green: https://ci.faas.to.cf-app.com/teams/pfs/pipelines/sk8s/jobs/run-sk8s-system-tests/builds/27

[freynca]

The following changes were needed in sk8s and surrounding projects, in order to pass end-to-end deployment and testing:

Note: these changes are currently applied in CI / system-test codebases; PRs should be issued to upstream repositories if appropriate.

Greeter Java Sample

• Add the following properties to the build

  <properties>
      <java.version>1.8</java.version>
          <maven.compiler.source>1.8</maven.compiler.source>
          <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

• Override function name, input and output topics with dynamically-generated values in k8s workload yaml

• Override docker image name/tag with dynamically generated value in k8s workload yaml

• Override docker image tag in sample Dockerfile

Helm Chart

• Change image pull policy from IfNotPresent to Always in values.yaml
• Change image tags for function-controller, zipkin-server, topic-controller, and http-gateway from hardcoded values to the currently deployed version
• Add missing SK8S_FUNCTION_CONTROLLER_SIDECAR_TAG env var to the function-controller deployment, pointing to the appropriate version of the sidecar image
• Remove liveness and readiness probes from the topic-controller deployment, as the Go topic-controller does not currently listen on a port
• Use the chart version hardcoded in Chart.yaml

Golang Components

• Change base Docker image from scratch to alpine on sidecar, http-gateway, and topic-controller, as the scratch version does not start up

[UNRESOLVED] Sidecar Build

• Test fails due to message not delivered in Kafka; still investigating...

  2017/11/16 14:25:05 Waiting for function to accept connection on localhost:8080
  --- FAIL: TestIntegrationWithKafka (0.89s)
  function-sidecar_test.go:86: kafka: Failed to deliver 1 messages.

  @markfisher @jldec FYI

[jldec]

@freynca what's the version number of that green build? Was a tag pushed to GitHub? Can i use a helm chart to deploy it?

[freynca]

@jldec The individual components have been reset to 0.0.1-SNAPSHOT.1 a few minutes ago. The overall Helm Chart version is still 0.0.1. A fresh full build is running now, and artifacts will be available shortly.

[freynca]

@jldec @markfisher

The following artifact is green for consumption:

Helm Chart: https://sk8s_charts.storage.googleapis.com/sk8s-0.0.1.tgz Version: 0.0.1 SHA256: 659bad47045d7e315bcc1d0b3f5f0f5a17ebbf737db4efd5ebf98689af5915dd Created: 2017-11-16 13:19 Eastern

Image References:

sk8s/function-controller:0.0.1-SNAPSHOT.1 [sha256:73d2e16ead650b57459854090ca6f4583707bac6a242751085c1e140ddf85773]
sk8s/topic-controller:0.0.1-SNAPSHOT.1 [sha256:00ec5e1ab5b6caf1a010433bc7cd7b18fd062bdb565a543d1f8caafc036584ab]
sk8s/http-gateway:0.0.1-SNAPSHOT.1 [sha256:d66a08387de5d5a05d1e68b2557a9895025954eeb78d439eff241373bb088b15]
sk8s/zipkin-server:0.0.1-SNAPSHOT.1 [sha256:65e032c967eb818fe53c6d07b561a96b46237c555f939f1d4200edb56ec05833]
sk8s/function-sidecar:0.0.1-SNAPSHOT.1 [sha256:5d96be93b96aadfaeebb4910b23efa6c085b7e6d72d4a94d9e1c8b19b1f8cd87]
sk8s/shell-function-invoker:0.0.1-SNAPSHOT.1 [sha256:cc23ace9c274aa442ec97be9ccd5000dc28be7933545cf2792e51fccd9b02c5b]
sk8s/java-function-invoker:0.0.1-SNAPSHOT.1 [sha256:2dab1294555598f0a58f7f1b68b8543418c3ce9f953c9dccbacd79fc84b19690]
sk8s/node-function-invoker::0.0.1-SNAPSHOT.1 [sha256:bc49089ee49b126fa36a6f9f315fa3c0a01c10e2de3265922590d54c7f84fe18]
grafana/grafana:4.4.1
prom/prometheus:v1.7.1
wurstmeister/zookeeper:3.4.6

[markfisher]

can we use namespaces instead of generating names for functions and topics? also at the function level is the image name generated or just the tag?

[freynca]

@markfisher Deployment and test execution is already namespaced w.r.t. k8s. We randomize the other constructs to avoid collisions/duplication with any existing objects, thus corrupting the outcome of the test. The generated function container is also test-specific (name and tag), for the same reason.

[markfisher]

what "existing objects" are you worried about having collisions with?

[freynca]

If the test is run multiple times, the function+topics would already exist in k8s. The function image for each run also needs to be uniquely identifiable (granted a shared name with a unique tag is sufficient for this).

[markfisher]

shouldn't the test clean up after itself, and shouldn't the tag be the unique identifier?

[freynca]

Fair points - I'll update the tests to perform cleanup on success, and tag the function container for uniqueness. However, these 2 changes are not core to the list above. I'm hoping this summary would lead to conversations to amend relevant codebases, where appropriate.

[markfisher]

RE:

Change image pull policy from IfNotPresent to Always in values.yaml

We want to keep the default as "IfNotPresent", but that is an overridable property, so the build could override if necessary.

[markfisher]

How are you building the Java sample function? The greeter sample in the sk8s repo doesn't have those properties you listed, and it builds fine (with the included ./mvnw).

[markfisher]

I've created this issue for addressing the FROM scratch Go images: https://github.com/markfisher/sk8s/issues/148

[markfisher]

regarding the liveness and readiness probes for the topic-controller, those will be updated when the deployment yaml is officially switching from the Java version. That hasn't happened yet (so we should stick with the Java topic-controller in CI until that does happen), but this is the issue to track: https://github.com/markfisher/sk8s/issues/145

[freynca]

• The sample is built using mvn clean package; will remove the extra properties if using the ./mvnw wrapper corrects the issue
• Switching to the Java topic controller now; re-adding liveness and readiness probes to the deployment

[freynca]

@jldec @markfisher

The following artifact is green for consumption:

Helm Chart: https://sk8s_charts.storage.googleapis.com/sk8s-0.0.1.tgz Version: 0.0.1 SHA256: c2209ad23d98290a6faaf7bf691d430b74ff4fc0b90f8dca0d652d55a05f4796 Created: 2017-11-17 11:35 Eastern

Image References:

sk8s/function-controller:0.0.1-SNAPSHOT.1 [sha256:c2209ad23d98290a6faaf7bf691d430b74ff4fc0b90f8dca0d652d55a05f4796]
sk8s/topic-controller:0.0.1-SNAPSHOT.1 [sha256:cef4d033684422b05b53e7d4ea46769240c3d237829208e88dafb90e370a0c30]
sk8s/http-gateway:0.0.1-SNAPSHOT.1 [sha256:d66a08387de5d5a05d1e68b2557a9895025954eeb78d439eff241373bb088b15]
sk8s/zipkin-server:0.0.1-SNAPSHOT.1 [sha256:66604c2d03669e4a4479ea09df47340ec5b085491826f4d442c26a05fe2ded73]
sk8s/function-sidecar:0.0.1-SNAPSHOT.1 [sha256:5d96be93b96aadfaeebb4910b23efa6c085b7e6d72d4a94d9e1c8b19b1f8cd87]
sk8s/shell-function-invoker:0.0.1-SNAPSHOT.1 [sha256:0ee8dc6f024bd454ea86100e1c7be90001a6299c5e7fe13c32ecd194456511f1]
sk8s/java-function-invoker:0.0.1-SNAPSHOT.1 [sha256:1e416df656ab125a8f8a86eed1b3f2c0b71ca5c4310c8563466797a943e1d56d]
sk8s/node-function-invoker::0.0.1-SNAPSHOT.1 [sha256:974377e4dd9f1a0ba93c7cad03af71441885d932d86118e498cc9df725504002]
grafana/grafana:4.4.1
prom/prometheus:v1.7.1
wurstmeister/zookeeper:3.4.6

[freynca]

Resolved issues:

• Removed the need to set Java 1.8 properties, due to the usage of themvnw wrapper
• Added back liveness and readiness probes, as well as the Java topic-controller
• Clean up created test workload after a successful completion of the test

[trisberg]

Still seems like we are just overwriting existing artifacts re-using versions/tags, I would think we we would want to increment the suffix for each build. Instead of reusing 0.0.1-SNAPSHOT.1 each build we should bump it to 0.0.1-SNAPSHOT.2 and next build to 0.0.1-SNAPSHOT.3 and so on.

[freynca]

@markfisher I agree with Thomas' suggestion re: image snapshot versioning (even if we're not bumping the Helm Chart itself). Thoughts?

[trisberg]

I still don't understand why we need to publish a new Chart each build when we could just use the original one and override the version/tags using --set options and instead maybe publish a script that includes the overrides for each build

[freynca]

@trisberg A new chart is published because it's a self-contained artifact, with correct image tags built-in. If the image tags inside the chart are left unattended, the consumer of the chart would have no way of knowing what to set them to, even if they use the --set arguments when installing. The published chart, together with associated images tags and checksums, represents the entire FaaS artifact we produce, and can track.

[markfisher]

I agree about the snapshot versioning as well. In fact, based on previous discussions, it's what I was expecting already. Perhaps we want to consider 0.0.1.build-2 as the format? (more literally describes what triggered the generation of the artifact)

[trisberg]

IMO the chart is a curated deployment artifact intended for deploying released versions of the system. Using it to deploy snapshot build should require the user to override versions. To make it easier to install and demo snapshot versions I would suggest each build create a script that has the helm install command including all necessary overrides.

[markfisher]

re: the chart - we also discussed that at length, and the idea was to override the values but to emit the command used by the build (including those overrides) as a build artifact

[freynca]

@markfisher I'm making the changes to produce the override script now. Would you consider a PR adding the following to the function-controller-deployment.yaml?

 - name: SK8S_FUNCTION_CONTROLLER_SIDECAR_TAG
   value: {{ .Values.functionController.sidecar.image.tag }}

This way, all tags can be overridden via Helm.

[trisberg]

Yes, that would be great, will keep an eye out for it ...

[trisberg]

sorry wrong button ;)

[freynca]

• Override script now published alongside Helm Chart (https://sk8s_charts.storage.googleapis.com/sk8s-0.0.1-install-example.sh)
• Chart no longer has tags baked in
• Image snapshot values automatically bumped at the end of a successful run of mainline

[freynca]

System tests are green end-to-end on both Kubo and GKE. Closing.