Error executing hook: error executing PreStart hook "InstallPlugins": error listing installed plugins: exit status 1

In #179, the ElasticSearch E2E tests failed while waiting for the document count update test.

W1215 20:16:41.891] ++ kubectl --namespace test-elasticsearchcluster-1513368700-4458 get pilot es-test-mixed-0 '-o=go-template={{.status.elasticsearch.documents}}'
I1215 20:16:41.991] <no value> is not a number
I1215 20:16:42.035] <no value> is not a number
I1215 20:16:42.036] TEST FAILURE: Elasticsearch pilot did not update the document count

The cluster-info dump shows the following warning event:

        {
            "name": "es-test-mixed-0.150090ddd0972acd",
            "namespace": "test-elasticsearchcluster-1513368700-4458",
            "selfLink": "/api/v1/namespaces/test-elasticsearchcluster-1513368700-4458/events/es-test-mixed-0.150090ddd0972acd",
            "uid": "4b5b0425-e1d4-11e7-b2b8-5254002dee38",
            "resourceVersion": "1203",
            "creationTimestamp": "2017-12-15T20:12:37Z",
            "InvolvedObject": {
                "Kind": "Pilot",
                "Namespace": "test-elasticsearchcluster-1513368700-4458",
                "Name": "es-test-mixed-0",
                "UID": "2a4cb261-e1d4-11e7-b67c-0242ac110007",
                "APIVersion": "navigator.jetstack.io",
                "ResourceVersion": "4",
                "FieldPath": ""
            },
            "Reason": "ErrExecHook",
            "Message": "Error executing hook: error executing PreStart hook \"InstallPlugins\": error listing installed plugins: exit status 1",
            "Source": {
                "Component": "generic-pilot",
                "Host": ""
            },
            "FirstTimestamp": "2017-12-15T20:12:37Z",
            "LastTimestamp": "2017-12-15T20:14:36Z",
            "Count": 12,
            "Type": "Warning"
        },

And the following stack trace error in the ES pod logs

I1215 20:12:35.795915      21 write_config.go:41] Copying config file from "/etc/pilot/elasticsearch/config/..129812_15_12_20_11_41.197450752/elasticsearch.yml" to "/usr/share/elasticsearch/config/elasticsearch.yml"
I1215 20:12:35.814086      21 hook.go:102] Executed PreStart hook "WriteConfig"
I1215 20:12:35.814230      21 hook.go:97] Executing PreStart hook 'InstallPlugins'
Exception in thread "main" ElasticsearchParseException[null-valued setting found for key [node.data] found at line number [1], column number [11]]
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeValue(XContentSettingsLoader.java:171)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:113)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:75)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:52)
    at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:50)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1043)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1032)
    at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100)
    at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:72)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
    at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
    at org.elasticsearch.cli.Command.main(Command.java:90)
    at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
I1215 20:12:37.158046      21 hook.go:99] Error executing PreStart hook "InstallPlugins": error listing installed plugins: exit status 1
I1215 20:12:37.158256      21 helpers.go:34] Setting lastTransitionTime for Pilot "es-test-mixed-0" condition "Started" to 2017-12-15 20:12:37.158243554 +0000 UTC m=+2.065465907
I1215 20:12:37.159366      21 event.go:218] Event(v1.ObjectReference{Kind:"Pilot", Namespace:"test-elasticsearchcluster-1513368700-4458", Name:"es-test-mixed-0", UID:"2a4cb261-e1d4-11e7-b67c-0242ac110007", APIVersion:"navigator.jetstack.io", ResourceVersion:"4", FieldPath:""}): type: 'Warning' reason: 'ErrExecHook' Error executing hook: error executing PreStart hook "InstallPlugins": error listing installed plugins: exit status 1
I1215 20:12:37.193196      21 controller.go:64] Finished syncing pilot "test-elasticsearchcluster-1513368700-4458/es-test-mixed-0" (1.400045406s)
I1215 20:12:37.193787      21 controller.go:52] Error syncing Pilot test-elasticsearchcluster-1513368700-4458/es-test-mixed-0, requeuing: error executing PreStart hook "InstallPlugins": error listing installed plugins: exit status 1
I1215 20:12:37.194396      21 controller.go:31] processed work item
I1215 20:12:37.204681      21 hook.go:94] Skipping already executed hook for %!q(func() string=0xed7eb0) in phase "PreStart"

https://jetstack-build-infra.appspot.com/build/jetstack-logs/pr-logs/pull/jetstack_navigator/179/navigator-e2e-v1-7/318/

The test failed in the same way on the 1.8 cluster:

https://jetstack-build-infra.appspot.com/build/jetstack-logs/pr-logs/pull/jetstack_navigator/179/navigator-e2e-v1-8/330/

I don't think this is related to the change in the PR, but I'll re-run to double check.

/kind bug

jetstack / navigator

Error executing hook: error executing PreStart hook "InstallPlugins": error listing installed plugins: exit status 1 #180