search

lucidworks / logstash-output-json_batch

Batching implementation of http output for logstash
Other
10 stars 3 forks source link

Include Manticore Client Timeout In Output Config #1

Open gsperry opened 8 years ago

gsperry commented 8 years ago

I added the following log command at line 107 to try to do some troubleshooting: @logger.warn("Request failure #{count}: #{exception.message}.")

I am repeatedly getting the following errors: Request failure 0: 10.94.185.120:8764 failed to respond. {:level=>:warn} Request failure 0: Read timed out. {:level=>:warn}

My understanding from your comment is that Manticore has some issues with managing stale connections. However, as I get more log entries into Solr, the read timeout becomes more common. I have expanded the cluster to 6 servers hosting 12 shards. Logstash seems to begin crashing at about 450M Solr entries. Based on my understanding from other people's implementations, I feel like we shouldn't be hitting a wall at such a small volume with that amount of hardware.

My thought was to add a timeout setting in the Logstash output config that would set the timeout in the Manticore client. I have hunted around to try to find the information to help myself, but alas I am not knowledgeable enough with Ruby or Logstash to do this. Any help is much appreciated.

jwestberg commented 8 years ago

@gsperry apologies for not noticing this issue until this time. I've been successfully indexing beyond 450m locally, I'll see if I can stress test this and see what is breaking in your case.