Overriding default index name causes LogStash::ConfigurationError

[LorenzoScebba]

Hi,

First of all, thanks for the docker image, it's so great!

I'm trying to override the default index name in the output file "30-output.conf"

At the moment, my docker-compose looks something like this:

# https://elk-docker.readthedocs.io/

elk:
  image: my.container.registry/elk:761
  volumes:
    - ./02-beats-input.conf:/etc/logstash/conf.d/02-beats-input.conf
    - ./output.conf:/etc/logstash/conf.d/30-output.conf
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
    - "8080:8080" # Http sink

The image name is the result of building the following Dockerfile

FROM sebp/elk:761

WORKDIR ${LOGSTASH_HOME}
RUN gosu logstash bin/logstash-plugin install logstash-input-http

And the following is the content of the 2 .conf files :

02-beats-input.conf

input {
  http {
    host => "0.0.0.0"
    port => 8080
    codec => json
  }
  beats {
      port => 5044
  }
}

output.conf

output {
    elasticsearch {
        hosts => ["localhost"]
        manage_template => false
        index => "logs-index-%{+YYYY.MM.dd}"
    }
}

Everything looks great to me but after starting the docker-compose i get the following error:

elk_1  | [2020-04-27T16:07:55,334][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.1"}
elk_1  | [2020-04-27T16:07:55,393][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"6e2c9e86-5ba7-4cb4-82df-1778050c3d10", :path=>"/opt/logstash/data/uuid"}
elk_1  | [2020-04-27T16:08:02,231][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\"
, \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:47:in `compile_imperative'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:55:in `compile_graph'", "/op
t/logstash/logstash-core/lib/logstash/compiler.rb:17:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:14:in `compile_sources'", "org/logstash/execution/AbstractPipeli
neExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/crea
te.rb:36:in `execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
elk_1  | [2020-04-27T16:08:02,828][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
elk_1  | [2020-04-27T16:08:07,546][INFO ][logstash.runner          ] Logstash shut down.

The same docker-compose works if i comment or remove the volume mount to the ./output.conf file.

I've tried to re-type the output.conf file by hand in case of any illegal characters but i had no luck.

Do you have any ideas?

Thanks!

[spujadas]

Strange. Could you download the original file (https://raw.githubusercontent.com/spujadas/elk-docker/master/logstash-conf/30-output.conf) and bind-mount it to see if it's an issue with the mount or with the file?

[LorenzoScebba]

With the original file, and also with the index name changed, it works!

I'll post here the 2 files just for comparison

not-working-output.conf

output {
    elasticsearch {
        hosts => ["localhost"]
        manage_template => false
        index => "logs-index-%{+YYYY.MM.dd}"
    }
}

working-output.conf

output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "logs-index-%{+YYYY.MM.dd}"
  }
}

The only differences are the number of spaces for the indentation used, 2 in the working one, 4 for the not working one.

I will close this as the problem has been solved.

Thanks!