[BUG]Failed to install template {:message=>"Failed to load default template for OpenSearch v2 with ECS disabled

[novicejava1]

Hi,

I am trying to do this logstash and opensearch setup. Here are the details of the docker-compose file that i am using.

opensearch and dasboard service docker-compose file.

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:2.3.0
    #image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    #image: opensearchproject/opensearch:latest
    image: opensearchproject/opensearch:2.3.0
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.3.0
    #image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

logstash oss

  version: '2.1'
services:
  logstash:
    #image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.3
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
    ports:
      - "5044:5044"
    volumes:
      - $PWD/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    networks:
      - opensearch-net
networks:
  opensearch-net:

and here is the logstash.conf file.

input { 
  stdin { } 
}

filter {

}

output {
  opensearch {
    hosts => ["https://opensearch_fqdn:9200/"]
    index => "testindexing"
    user => "admin"
    password => "admin"
    ssl => true
    ssl_certificate_verification => false
   }
}

I am expecting that the testingindexing would be be created with the data that is sent to the stdin terminal. But i get the below error.

Error

logstash_1  | [2022-10-25T07:44:39,946][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Failed to load default template for OpenSearch v2 with ECS disabled; caused by: #<ArgumentError: Template file '/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json' could not be found>", :exception=>RuntimeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:33:in `load_default_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:21:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:412:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:247:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:224:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}

As per the below issue details - https://github.com/opensearch-project/opensearch-devops/issues/85, i tried to use the opensearchproject/logstash-oss-with-opensearch-output-plugin:8.4.0 for the logstash oss output plugin but the logstash terminates without much information.

logstash_1  | [2022-10-25T12:03:11,405][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["https://opensearch_fqdn:9200/"]}
logstash_1  | [2022-10-25T12:03:11,435][WARN ][logstash.outputs.opensearch][main] ** WARNING ** Detected UNSAFE options in opensearch output configuration!
logstash_1  | ** WARNING ** You have enabled encryption but DISABLED certificate verification.
logstash_1  | ** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
logstash_1  | [2022-10-25T12:03:11,691][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:xxxxxx@opensearch_fqdn:9200/]}}
logstash_1  | [2022-10-25T12:03:11,903][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://admin:xxxxxx@opensearch_fqdn:9200/"}
logstash_1  | [2022-10-25T12:03:11,956][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.3.0) {:version=>2}
logstash_1  | [2022-10-25T12:03:12,039][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:v8}
logstash_1  | [2022-10-25T12:03:12,058][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x50914657 run>"}
logstash_1  | [2022-10-25T12:03:12,571][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.51}
logstash_1  | [2022-10-25T12:03:12,622][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1  | [2022-10-25T12:03:12,692][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1  | [2022-10-25T12:03:13,008][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
logstash_1  | [2022-10-25T12:03:13,262][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
logstash_1  | [2022-10-25T12:03:13,360][INFO ][logstash.runner          ] Logstash shut down.

[deepdatta]

Hi @novicejava1, thanks for trying out logstash-output-opensearch. You were right about trying logstash-oss-with-opensearch-output-plugin:8.4.0 that should address the 'Failed to load default template..." issue. I tried it out from the docker image along with OpenSearch 2.3 deployed via docker compose but wasn't able to reproduce this issue. Can you please list the steps you used to deploy and configure the logstash container, it'll helm me figure what's causing the spontaneous shutdown.

[dblock]

Closing as logstash-oss-with-opensearch-output-plugin:8.4.0 should fix it. If someone is still experiencing this error, reopen.