[Error] Logstash 8.11.3 pipeline.conf (logstash.conf) issue

[F9Alejandro]

When trying to run the logstash portion of the compose it throws an error about the formatting of the .conf file. I am using the file that is provided in this repo and haven't made any changes to it.

Logstash error:

elastic-stack-logstash01-1    | [2023-12-23T05:23:21,079][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
elastic-stack-es01-1          | {"@timestamp":"2023-12-23T05:22:36.930Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-eck"}
elastic-stack-logstash01-1    | [2023-12-23T05:23:21,097][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
elastic-stack-logstash01-1    | [2023-12-23T05:23:21,104][INFO ][logstash.runner          ] Logstash shut down.
elastic-stack-logstash01-1    | [2023-12-23T05:23:21,107][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
elastic-stack-logstash01-1    | org.jruby.exceptions.SystemExit: (SystemExit) exit
elastic-stack-logstash01-1    |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
elastic-stack-logstash01-1    |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
elastic-stack-logstash01-1    |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

Any recommendations for trying to fix this issue?

[gk-f]

I've figured it out, logstash needs to know where is the pipeline .conf file, relevant part from docker-compose.yml:

  logstash01:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - certs:/usr/share/logstash/certs
      - logstashdata01:/usr/share/logstash/data
      - "./logstash_ingest_data/:/usr/share/logstash/ingest_data/"
      - "./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro"
    environment:
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
    command:
      - -f
      - /usr/share/logstash/pipeline/

Add the command part to the code.

[elkninja]

thanks for the heads up. i'll test the one out.

[elkninja]

i am unable to recreate this error and do not need to specify the -f command as this is taken care of by passing in the logstash.conf to the default pipeline directory.

[gk-f]

@elkninja probably because this issue is related to version 8.11.3 and above. I had the same error message with 8.12.0.