search

logsearch / logsearch-boshrelease

A BOSH-scalable Elasticsearch+Logstash+Kibana release
http://www.logsearch.io
Apache License 2.0
57 stars 46 forks source link

Upgrade elasticsearch past 1.2.1 #91

Closed dpb587 closed 9 years ago

dpb587 commented 10 years ago

Current stable is 1.4.0

dpb587 commented 10 years ago

Currently the sonian/elasticsearch-jetty we're using for request logging is not compatible with 1.4.0. Issue sonian/elasticsearch-jetty#69 documents it; our exact error is:

{1.4.0}: Initialization Failed ...
1) Could not find a suitable constructor in com.sonian.elasticsearch.http.filter.FilterHttpServerTransportModule. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.

I've pushed the WIP to elasticsearch-1-4-0 (with sonian disabled). Other than sonian, things seem to be working fine. I tested it on bosh-lite and also deployed it to one of my micro deployments where I don't care about the requests.log performance.

I also went through the _upgrade process on a couple of my logstash-* indices and they seemed to work successfully. I did run into errors trying to run the _upgrade globally (with ~100 indices, ~4 shards each):

Caused by: java.io.FileNotFoundException: /var/vcap/store/elasticsearch/logsearch/nodes/0/indices/logstash-2014.10.18/2/index/_8o_es090_0.tip (Too many open files)

If we choose to upgrade our old indices on actual clusters, we should probably consider doing them systematically, sequentially instead of relying on elasticsearch to sanely queue itself. Alternatively, we not bother upgrading older indices in favor of just letting them get phased/curated out over time.

I'm not sure what step to take next. Presumably we still want the requests.log functionality that sonian provides (I don't think es/1.4 adds anything like that), otherwise we could just drop that. Semi-related, instead of trying to maintain it within the boshrelease (and marvel, elastichq, kibana), we could simply drop all the plugins and provide some sort of config option which will run all the bin/plugin -i {config-value-which-is-a-public-or-internal-urlish}. In this way we could proceed with 1.4.0 usage with a simpler path to re-integrate request logging when sonian/another plugin adds that support.

mrdavidlaing commented 10 years ago

Lets not bother with the index upgrade process.

I think its more important to get the stability benefits of 1.4.0 than keep the sonian request logs.

:+1: for the configurable plugin idea