Unable to create template on open search 1.3 getting 400 error

[gaddehyd]

Unable create template on open search 1.3 version using logstash 8.13.2 version. Getting below error, Opensearch is hosted on aws.

[Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:77] opensearch - Failed to install template {:message=>"Got response code '400' contacting OpenSearch at URL 'https://xxxx:443/_template/logstash'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:181:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:272:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:259:in block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:348:inwith_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:258:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:266:inblock in Pool'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:404:in template_put'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:82:intemplate_install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/template_manager.rb:37:in install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/template_manager.rb:25:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:419:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:254:infinish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:231:in block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:inblock in after_successful_connection'"]}

[dblock]

This is similar to https://github.com/opensearch-project/logstash-output-opensearch/issues/227, https://github.com/opensearch-project/logstash-output-opensearch/issues/179, https://github.com/opensearch-project/logstash-output-opensearch/issues/176, and possibly others. The generic 400 error is unfortunately not helpful. Is there more in the output? Enable --debug, can we get the body of the error and whatever is around this error in the logstash log?

I opened https://github.com/opensearch-project/logstash-output-opensearch/issues/251, maybe you/someone can help improve this?

[gaddehyd]

@dblock It's a odd behaviour for my case, after adding --debug I am able to create template but with logstash -f file_name getting 400 error

[dblock]

@gaddehyd that would definitely be a bug! maybe some code that only gets executed on debug? what are the versions that you're using? post relevant logs with and without?

[gaddehyd]

logstash-with-debug.txt logstash-without-debug.txt @dblock I tried on logstash 8.13.2 and 7.15.0 versions and Open Search plugin 2.0.2

[dblock]

Thanks. This has to be a red herring and a timing problem where by enabling debug logs we end up doing something out of order. There's nothing I found in the logs out of ordinary.

Are you able to build and use the plugin code here and make some changes? I would make a change in https://github.com/opensearch-project/logstash-output-opensearch/blob/f45b8c753ac4366e91194c682c85ced3858c9792/lib/logstash/outputs/opensearch/http_client/pool.rb#L26 to display the body of the error to start.

I have details of how to do the above in https://github.com/opensearch-project/logstash-output-opensearch/issues/186#issuecomment-1941707311 (and #250 to add this to the dev guide).

[gaddehyd]

[2024-04-29T05:59:12,764][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Got response code '400' contacting OpenSearch at URL 'https://url:443/_template/template_toptracks_stats'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:181:in perform_request'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client/pool.rb:272:inperform_request_to_url'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client/pool.rb:259:in block in perform_request'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client/pool.rb:348:inwith_connection'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client/pool.rb:258:in perform_request'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client/pool.rb:266:inblock in Pool'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client.rb:404:in template_put'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/http_client.rb:82:intemplate_install'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/template_manager.rb:37:in install'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch/template_manager.rb:25:ininstall_template'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch.rb:419:in install_template'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch.rb:254:infinish_register'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/outputs/opensearch.rb:231:in block in register'", "/usr/share/logstash/vendor/local_gems/c5127f68/logstash-output-opensearch-2.0.2-universal.arm64e-darwin-23/lib/logstash/plugin_mixins/opensearch/common.rb:83:inblock in after_successful_connection'"]}

@dblock Again same error, I have build the plugin and deployed on logstash. I am running image on AWS Batch service.

[dblock]

@gaddehyd the code should be modified to display the @response_body, this message is unfortunately not helpful

[gaddehyd]

@dblock Can you suggest where I need to update the code for response_body ? I have searched with log string which you are updated last time in code.

[dblock]

Here: https://github.com/opensearch-project/logstash-output-opensearch/blob/f45b8c753ac4366e91194c682c85ced3858c9792/lib/logstash/outputs/opensearch/http_client/pool.rb#L26C86-L27C10

It says

"Got response code '#{response_code}' contacting OpenSearch at URL '#{@url}'"

try changing it to

"Got response code '#{response_code}' contacting OpenSearch at URL '#{@url}, request: #{@request_body}, response: #{@response_body}'"

I can make some time to do this IRL if you want on zoom, find me on the public slack, I'm @ dblock.

[gaddehyd]

@dblock After adding shared lines of code found below error response: {\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [varchar] declared on field [album_title as album]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [varchar] declared on field [title as album]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [varchar] declared on field [title as album]\"}},\"status\":400}'",

After changing the data type varchar to text, Issue got resolved

[dblock]

This is helpful! The error seems common, SO says that string was dropped in favor of text or uuid at some point a long time ago.

So what's your initial state? What is in this album index? Do you think you can reproduce this from a totally clean setup with real simple data and instructions that anyone can follow?

[gaddehyd]

@dblock I have updated my index template with open search supported data types(https://opensearch.org/docs/latest/search-plugins/sql/datatypes/), But we getting new issue after updating the template. response: {\"error\":{\"root_cause\":[{\"type\":\"not_x_content_exception\",\"reason\":\"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes\"}],\"type\":\"not_x_content_exception\",\"reason\":\"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes\"},\"status\":500}'"

[dblock]

I still would like a detailed repro so that we can actually improve the code to show the error here. Help us build one? Something super rudimentary with 5 steps from scratch would be amazing.

I have updated my index template with open search supported data types(https://opensearch.org/docs/latest/search-plugins/sql/datatypes/)

Good. This is one less problem. Could you please give a tl;dr on exactly what you had and did for the next person after you have repro steps?

Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes

Where's this error from? Same place? I am not familiar with this error, but there's a lot of google hits that say something about the data format being the problem / json being sent with the wrong content type, etc.