Error 400

[grigio]

Hi, I tried this plugin on the same pc and it works fine, but now I specified an external host and it seems the request are forwarded but the receiver server respond with error 400.

It could be something like:

400 Bad Request
'json' or 'msgpack' parameter is required

# nginx log:
79.16.x.x - - [24/Jul/2013:22:07:05 +0200] "POST /_bulk HTTP/1.1" 400 58 "-" "curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

[pitr]

can you share your configurations?

[grigio]

Yes

<source>
  type syslog
  port 42185
  tag syslog
</source>

<source>
  type forward
</source>

<match **.**>
  type copy
  <store>
    type stdout #debug
  </store>
  <store>
    type elasticsearch

    host elasticsearchapi.example.com
    port 80

    include_tag_key true
    tag_key _key
    logstash_format true
    flush_interval 10s # for testing
  </store>
</match>

I see the correct json on the fluentd client stdout and I get the "POST /_bulk HTTP/1.1" 400" request on the public elastic search api. I've NGINX in front of it and I know it works because I'm already collecting data via simple http request:

..
curl http://elasticsearchapi.example.com/namespace.foo -X POST -d 'json={\"ip\":\"#{@ip}\",\"time_elapsed\":#{@time_elapsed},\"message\":\"#{@e || ''}\"}'"
..

[pitr]

let me understand your setup correctly. You have FluentD setup to listen on TCP and Syslog and write to stdout and ElasticSearch. Where does nginx come in here? Does it log to syslog? Or is it located in-between FluentD and ElasticSearch?

[grigio]

nginx is between Fluend and ElasticSearch.

remote pc with (Fluentd/syslog) -> elasticsearchapi.example.com:80 (nginx) -> localhost:9200 (ES)

In a local machine without the "host" parameter and nginx it collected the data

[roughley]

I've noticed the same behaviour - ip address forwarded correctly to es (no nginx) but using a hostname didn't.

Haven't had the time to dig into more, but its an issue because we need to use a hostname than can balance over a cluster.

Luigi Maselli notifications@github.com wrote:

nginx is between Fluend and ElasticSearch.

remote pc with (Fluentd/syslog) -> elasticsearchapi.example.com:80 (nginx) -> localhost:9200 (ES)

In a local machine without the "host" parameter and nginx it collected the data

— Reply to this email directly or view it on GitHub.

[grigio]

I think the bug is in the net/http POST request, because It doesn't seem the referer is set somewhere

[grigio]

The tests are ok, but it doesn't seem to fix this problem.. ideas?

2013-07-29 15:13:47 +0200 fino.asl: {"event":"event-123","resp_time":5}
2013-07-29 15:13:56 +0200 [warn]: temporarily failed to flush the buffer. next_retry=2013-07-29 15:15:03 +0200 error_class="Net::HTTPServerException" error="400 \"Bad Request\"" instance=70096177534460
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:2632:in `error!'
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:2641:in `value'
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/gems/ruby-1.9.3-p194/gems/fluent-plugin-elasticsearch-0.1.3/lib/fluent/plugin/out_elasticsearch.rb:67:in `write'
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/gems/ruby-1.9.3-p194/gems/fluentd-0.10.35/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/gems/ruby-1.9.3-p194/gems/fluentd-0.10.35/lib/fluent/buffer.rb:274:in `pop'
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/gems/ruby-1.9.3-p194/gems/fluentd-0.10.35/lib/fluent/output.rb:294:in `try_flush'
  2013-07-29 15:13:56 +0200 [warn]: /Users/grigio/.rvm/gems/ruby-1.9.3-p194/gems/fluentd-0.10.35/lib/fluent/output.rb:121:in `run'
2013-07-29 15:13:56 +0200 fluent.warn: {"next_retry":"2013-07-29 15:15:03 +0200","error_class":"Net::HTTPServerException","error":"400 \"Bad Request\"","instance":70096177534460,"message":"temporarily failed to flush the buffer."}

I don't know, on Linux I don't have error 400 on the client but the results doesn't change

[pitr]

I am not clear as to where the 400 error comes from. The only reference of 'json' or 'msgpack' parameter is required I can find is in Fluent::HttpInput - https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_http.rb#L115

[pitr]

@grigio @roughley any update on this? I'm closing this otherwise

[roughley]

I haven't looked into it further, and setup an internal proxy to round-robin the requests for HA. If we decide to setup an internal DNS, I'll take another look.

☃ pitr wrote:

@grigio https://github.com/grigio @roughley https://github.com/roughley any update on this? I'm closing this otherwise

— Reply to this email directly or view it on GitHub https://github.com/uken/fluent-plugin-elasticsearch/issues/8#issuecomment-22970244.