Multiple pipelines with xpack management not working

[jordansissel]

I have Logstash in Kuberenetes (abbreviated config shown):

      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:6.2.3
        env:
          - name: XPACK_MANAGEMENT_PIPELINE_ID
            value: "infosec-cloudtrail"
          - name: XPACK_MANAGEMENT_ENABLED
            value: "true"

When executing, this creates a logstash.yml with the following (observed with kubectl exec logstash ...):

xpack.management.pipeline.id: infosec-cloudtrail,beats

This seems incorrect, I think it should use a list in YAML syntax:

xpack.management.pipeline.id: [ "infosec-cloudtrail", "beats" ]

I don't know if we have any automated way of telling env2yaml that a given setting is an array.

---

I'm going to work around this by shipping my own logstash.yml (mainly because it'll be simpler in k8s because every ENV var in k8s is minimum 2 yaml lines (often 5 if using a secret).

[ninaspitfire]

It looks like it works if you pass a string that is a valid YAML array.

$ IMAGE=docker.elastic.co/logstash/logstash:6.2.3
$ CMD='cat /usr/share/logstash/config/logstash.yml'
$ docker run \
>   -e xpack.management.enabled=true \
>   -e xpack.management.pipeline.id='[custom,extra]' \
>   $IMAGE $CMD

2018/11/06 00:11:34 Setting 'xpack.management.enabled' from environment.
2018/11/06 00:11:34 Setting 'xpack.management.pipeline.id' from environment.
http.host: 0.0.0.0
xpack.management.enabled: true
xpack.management.pipeline.id:
- custom
- extra
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.elasticsearch.url: http://elasticsearch:9200
xpack.monitoring.elasticsearch.username: logstash_system

I'm still confused about where the beats pipeline ID came from, though.

EDIT: Here's that same config in JSON, just to be explicit:

{
  "http.host": "0.0.0.0",
  "xpack.management.enabled": true,
  "xpack.management.pipeline.id": [
    "custom",
    "extra"
  ],
  "xpack.monitoring.elasticsearch.password": "changeme",
  "xpack.monitoring.elasticsearch.url": "http://elasticsearch:9200",
  "xpack.monitoring.elasticsearch.username": "logstash_system"
}

[a1exus]

docker-compose:

# grep -i pipeline.id docker-compose.override.yml 
                        - XPACK_MANAGEMENT_PIPELINE_ID=["pipelineA","pipelineB","pipelineC","pipelineD","pipelineE","pipelineF"]
#