WIP: Notes and modifications to allow testing ES pilot in isolation

[wallrj]

@munnerz Here's a quick dump of the changes I made and the commands I ran to test the es pilot from my laptop.

And actually, I find that the pilot does shutdown after sigterm, so long as I've added the xpack.security.enabled: false config line.

Release note:

NONE

[jetstack-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: We suggest the following additional approver: munnerz

Assign the PR to them by writing /assign @munnerz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files: - **[OWNERS](https://github.com/jetstack/navigator/blob/master/OWNERS)** You can indicate your approval by writing `/approve` in a comment You can cancel your approval by writing `/approve cancel` in a comment

[wallrj]

Some errors spotted as it shut down....which might be interesting:

I1214 15:43:30.559131       1 genericpilot.go:94] Shutdown signal received
I1214 15:43:30.559155       1 genericpilot.go:97] Waiting for process exit and hooks to execute
I1214 15:43:36.370038       1 event.go:218] Event(v1.ObjectReference{Kind:"Pilot", Namespace:"foo", Name:"bar", UID:"d0303404-e0df-11e7-ba2b-0a580a20171a", APIVersion:"navigator.jetstack.io", ResourceVersion:"76", FieldPath:""}): type: 'Normal' reason: 'ExecHook' Completed phase "PreStop"
[2017-12-14T15:43:36,372][INFO ][o.e.n.Node               ] [KCzdJA6] stopping ...
[2017-12-14T15:43:36,393][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/87] [Main.cc@168] Ml controller exiting
[2017-12-14T15:43:36,394][INFO ][o.e.x.w.w.WatchStore     ] [KCzdJA6] stopped watch store
[2017-12-14T15:43:36,394][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
[2017-12-14T15:43:37,348][INFO ][o.e.n.Node               ] [KCzdJA6] stopped
[2017-12-14T15:43:37,348][INFO ][o.e.n.Node               ] [KCzdJA6] closing ...
[2017-12-14T15:43:37,358][INFO ][o.e.n.Node               ] [KCzdJA6] closed
I1214 15:43:37.453494       1 helpers.go:41] Found status change for Pilot "bar" condition "Started": "True" -> "False"; setting lastTransitionTime to 2017-12-14 15:43:37.453476642 +0000 UTC m=+52.407094825
I1214 15:43:37.480476       1 controller.go:64] Finished syncing pilot "foo/bar" (1.110665739s)
I1214 15:43:37.480555       1 controller.go:52] Error syncing Pilot foo/bar, requeuing: exit status 129
I1214 15:43:37.480656       1 controller.go:31] processed work item
I1214 15:43:37.481463       1 event.go:218] Event(v1.ObjectReference{Kind:"Pilot", Namespace:"foo", Name:"bar", UID:"d0303404-e0df-11e7-ba2b-0a580a20171a", APIVersion:"navigator.jetstack.io", ResourceVersion:"76", FieldPath:""}): type: 'Normal' reason: 'ExecHook' Completed phase "PostStop"
I1214 15:43:37.530435       1 controller.go:64] Finished syncing pilot "foo/bar" (49.726322ms)
I1214 15:43:37.530504       1 controller.go:52] Error syncing Pilot foo/bar, requeuing: [Get http://172.17.0.2:9200/_nodes/_local/stats: dial tcp 172.17.0.2:9200: getsockopt: connection refused, Operation cannot be fulfilled on pilots.navigator.jetstack.io "bar": the object has been modified; please apply your changes to the latest version and try again]
I1214 15:43:37.530651       1 controller.go:31] processed work item
I1214 15:43:37.533059       1 controller.go:175] No phase hooks to execute
I1214 15:43:37.559373       1 genericpilot.go:102] Shutting down workqueue
I1214 15:43:37.559456       1 controller.go:64] Finished syncing pilot "foo/bar" (26.388579ms)
I1214 15:43:37.559517       1 controller.go:52] Error syncing Pilot foo/bar, requeuing: [no available connection: no Elasticsearch node available, Operation cannot be fulfilled on pilots.navigator.jetstack.io "bar": the object has been modified; please apply your changes to the latest version and try again]
I1214 15:43:37.559595       1 controller.go:31] processed work item
I1214 15:43:37.559627       1 controller.go:33] exiting worker loop
I1214 15:43:37.559479       1 genericpilot.go:105] Shutting down generic pilot controller workers
I1214 15:43:37.559685       1 genericpilot.go:108] Generic pilot controller workers stopped

[jetstack-ci-bot]

@wallrj PR needs rebase

[munnerz]

@wallrj I'm going to close this on account of not wanting to encourage manually running Pilots (i.e. not creating Pilots via Navigator). Re-open if you disagree. I think running a local Kubernetes cluster (either via Docker for Mac or minikube or the like) should be how developers test changes to Navigator.