search

salyh / elasticsearch-imap

IMAP and POP3 email importer for Elasticsearch (no river anymore)
Apache License 2.0
100 stars 25 forks source link

River block ES cluster restart (reported by edgar by mail) #18

Closed salyh closed 9 years ago

salyh commented 9 years ago

this only happened for ES >= 1.4, seems to be a deadlock issue in ES 1.4, changed initialization to be more safe here, hopefully 7fb8b03 solves this issue also for ES 1.4

salyh commented 9 years ago

fixed

deancsmith commented 9 years ago

I've been having the above problem (I believe) still, both on ES 1.4 and ES 1.5 running on Windows.

River-IMAP, once configured, seems to cause ES to hang when restarting.

Sorry, I don't have any logs as I've downgraded to ES 1.3.x to resolve the issue.

salyh commented 9 years ago

I have no windows but i ll look into this

Am 27.05.2015 um 13:41 schrieb Dean Smith notifications@github.com:

I've been having the above problem (I believe) still, both on ES 1.4 and ES 1.5 running on Windows.

River-IMAP, once configured, seems to cause ES to hang when restarting.

Sorry, I don't have any logs as I've downgraded to ES 1.3.x to resolve the issue.

— Reply to this email directly or view it on GitHub.

deancsmith commented 9 years ago

Thanks, I don't think that Windows is the issue, but I thought I'd mention it.

If you can't replicate with your setup, then I'll find some time to set up a linux box configured for my use case and see if I can see the same issue.

deancsmith commented 9 years ago

I found some of my logs. This is from ES 1.4.5, where I think River-IMAP was also causing ES to hang when starting to poll an inbox:

[2015-05-27 19:55:41,608][DEBUG][plugins                  ] [Trick Shot] [C:\apps\elasticsearch\plugins\river-imap\_site] directory does not exist.
[2015-05-27 19:58:21,131][DEBUG][plugins                  ] [Trick Shot] [C:\apps\elasticsearch\plugins\river-imap\_site] directory does not exist.
[2015-05-27 19:58:39,987][DEBUG][cluster.service          ] [Trick Shot] processing [create-index [_river], cause [auto(index api)]]: execute
[2015-05-27 19:58:39,993][DEBUG][indices                  ] [Trick Shot] creating Index [_river], shards [1]/[1]
[2015-05-27 19:58:40,490][DEBUG][index.mapper             ] [Trick Shot] [_river] using dynamic[true], default mapping: default_mapping_location[null], loaded_from[jar:file:/C:/apps/elasticsearch/lib/elasticsearch-1.4.5.jar!/org/elasticsearch/index/mapper/default-mapping.json], default percolator mapping: location[null], loaded_from[null]
[2015-05-27 19:58:40,490][DEBUG][index.cache.query.parser.resident] [Trick Shot] [_river] using [resident] query cache with max_size [100], expire [null]
[2015-05-27 19:58:40,490][DEBUG][index.store.fs           ] [Trick Shot] [_river] using index.store.throttle.type [node], with index.store.throttle.max_bytes_per_sec [0b]
[2015-05-27 19:58:40,671][INFO ][cluster.metadata         ] [Trick Shot] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings [nc-search]
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing ... (reason [cleaning up after validating index on master])
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing index service
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing index cache
[2015-05-27 19:58:40,686][DEBUG][index.cache.filter.weighted] [Trick Shot] [_river] full cache clear, reason [close]
[2015-05-27 19:58:40,686][DEBUG][index.cache.fixedbitset  ] [Trick Shot] [_river] Clearing all FixedBitSets because [close]
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] clearing index field data
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing analysis service
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing index engine
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing index gateway
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing mapper service
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing index query parser service
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closing index service
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] [_river] closed... (reason [cleaning up after validating index on master])
[2015-05-27 19:58:40,686][DEBUG][cluster.service          ] [Trick Shot] cluster state updated, version [3], source [create-index [_river], cause [auto(index api)]]
[2015-05-27 19:58:40,686][DEBUG][cluster.service          ] [Trick Shot] publishing cluster state version 3
[2015-05-27 19:58:40,686][DEBUG][cluster.service          ] [Trick Shot] set local cluster state to version 3
[2015-05-27 19:58:40,686][DEBUG][indices.cluster          ] [Trick Shot] [_river] creating index
[2015-05-27 19:58:40,686][DEBUG][indices                  ] [Trick Shot] creating Index [_river], shards [1]/[1]
[2015-05-27 19:58:40,746][DEBUG][index.mapper             ] [Trick Shot] [_river] using dynamic[true], default mapping: default_mapping_location[null], loaded_from[jar:file:/C:/apps/elasticsearch/lib/elasticsearch-1.4.5.jar!/org/elasticsearch/index/mapper/default-mapping.json], default percolator mapping: location[null], loaded_from[null]
[2015-05-27 19:58:40,746][DEBUG][index.cache.query.parser.resident] [Trick Shot] [_river] using [resident] query cache with max_size [100], expire [null]
[2015-05-27 19:58:40,746][DEBUG][index.store.fs           ] [Trick Shot] [_river] using index.store.throttle.type [node], with index.store.throttle.max_bytes_per_sec [0b]
[2015-05-27 19:58:40,746][DEBUG][indices.cluster          ] [Trick Shot] [_river] adding mapping [nc-search], source [{"nc-search":{"properties":{}}}]
[2015-05-27 19:58:40,746][DEBUG][indices.cluster          ] [Trick Shot] [_river][0] creating shard
[2015-05-27 19:58:40,746][DEBUG][index.service            ] [Trick Shot] [_river] creating shard_id [0]
[2015-05-27 19:58:40,972][DEBUG][index.store              ] [Trick Shot] [_river][0] store stats are refreshed with refresh_interval [10s]
[2015-05-27 19:58:40,972][DEBUG][index.deletionpolicy     ] [Trick Shot] [_river][0] Using [keep_only_last] deletion policy
[2015-05-27 19:58:40,972][DEBUG][index.merge.policy       ] [Trick Shot] [_river][0] using [tiered] merge mergePolicy with expunge_deletes_allowed[10.0], floor_segment[2mb], max_merge_at_once[10], max_merge_at_once_explicit[30], max_merged_segment[5gb], segments_per_tier[10.0], reclaim_deletes_weight[2.0]
[2015-05-27 19:58:40,972][DEBUG][index.merge.scheduler    ] [Trick Shot] [_river][0] using [concurrent] merge scheduler with max_thread_count[1], max_merge_count[3]
[2015-05-27 19:58:40,972][DEBUG][index.shard.service      ] [Trick Shot] [_river][0] state: [CREATED]
[2015-05-27 19:58:40,972][DEBUG][index.shard.service      ] [Trick Shot] [_river][0] scheduling optimizer / merger every 1s
[2015-05-27 19:58:40,987][DEBUG][index.translog           ] [Trick Shot] [_river][0] interval [5s], flush_threshold_ops [2147483647], flush_threshold_size [200mb], flush_threshold_period [30m]
[2015-05-27 19:58:40,987][DEBUG][index.shard.service      ] [Trick Shot] [_river][0] state: [CREATED]->[RECOVERING], reason [from gateway]
[2015-05-27 19:58:40,987][DEBUG][index.gateway            ] [Trick Shot] [_river][0] starting recovery from local ...
[2015-05-27 19:58:41,008][DEBUG][cluster.service          ] [Trick Shot] processing [create-index [_river], cause [auto(index api)]]: done applying updated cluster_state (version: 3)
[2015-05-27 19:58:40,987][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: execute
[2015-05-27 19:58:41,036][DEBUG][index.engine.internal    ] [Trick Shot] [_river][0] starting engine
[2015-05-27 19:58:41,051][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-05-27 19:58:41,136][DEBUG][index.shard.service      ] [Trick Shot] [_river][0] scheduling refresher every 1s
[2015-05-27 19:58:41,137][DEBUG][index.shard.service      ] [Trick Shot] [_river][0] state: [RECOVERING]->[POST_RECOVERY], reason [post recovery from gateway, no translog]
[2015-05-27 19:58:41,137][DEBUG][index.gateway            ] [Trick Shot] [_river][0] recovery completed from [local], took [150ms]
[2015-05-27 19:58:41,137][DEBUG][cluster.action.shard     ] [Trick Shot] sending shard started for [_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], reason [after recovery from gateway]
[2015-05-27 19:58:41,138][DEBUG][cluster.action.shard     ] [Trick Shot] received shard started for [_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], reason [after recovery from gateway]
[2015-05-27 19:58:41,138][DEBUG][cluster.service          ] [Trick Shot] processing [shard-started ([_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING]), reason [after recovery from gateway]]: execute
[2015-05-27 19:58:41,139][DEBUG][cluster.action.shard     ] [Trick Shot] [_river][0] will apply shard started [_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], reason [after recovery from gateway]
[2015-05-27 19:58:41,141][DEBUG][cluster.service          ] [Trick Shot] cluster state updated, version [4], source [shard-started ([_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING]), reason [after recovery from gateway]]
[2015-05-27 19:58:41,141][DEBUG][cluster.service          ] [Trick Shot] publishing cluster state version 4
[2015-05-27 19:58:41,141][DEBUG][cluster.service          ] [Trick Shot] set local cluster state to version 4
[2015-05-27 19:58:41,143][DEBUG][index.shard.service      ] [Trick Shot] [_river][0] state: [POST_RECOVERY]->[STARTED], reason [global state is [STARTED]]
[2015-05-27 19:58:41,144][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: execute
[2015-05-27 19:58:41,148][DEBUG][cluster.service          ] [Trick Shot] processing [shard-started ([_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING]), reason [after recovery from gateway]]: done applying updated cluster_state (version: 4)
[2015-05-27 19:58:41,164][DEBUG][river.routing            ] [Trick Shot] no river _meta document found retrying in 1000 ms
[2015-05-27 19:58:41,164][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-05-27 19:58:41,195][DEBUG][cluster.service          ] [Trick Shot] processing [update-mapping [_river][nc-search] / node [5izIMXpCQVWKMV4L9gI9rw], order [1]]: execute
[2015-05-27 19:58:41,210][DEBUG][cluster.metadata         ] [Trick Shot] [_river] update_mapping [nc-search] (dynamic) with source [{"nc-search":{"properties":{"bulk_flush_interval":{"type":"string"},"bulk_size":{"type":"long"},"interval":{"type":"string"},"keep_expunged_messages":{"type":"boolean"},"ldap_name_field":{"type":"string"},"mail.debug":{"type":"string"},"mail.imap.connectionpoolsize":{"type":"string"},"mail.imap.host":{"type":"string"},"mail.imap.port":{"type":"long"},"mail.imap.ssl.enable":{"type":"boolean"},"mail.imap.timeout":{"type":"long"},"mail.store.protocol":{"type":"string"},"mail_index_name":{"type":"string"},"mail_type_name":{"type":"string"},"max_bulk_requests":{"type":"string"},"password":{"type":"string"},"threads":{"type":"long"},"type":{"type":"string"},"user":{"type":"string"},"with_attachments":{"type":"boolean"},"with_flag_sync":{"type":"boolean"},"with_striptags_from_textcontent":{"type":"boolean"},"with_text_content":{"type":"boolean"}}}}]
[2015-05-27 19:58:41,268][DEBUG][cluster.service          ] [Trick Shot] cluster state updated, version [5], source [update-mapping [_river][nc-search] / node [5izIMXpCQVWKMV4L9gI9rw], order [1]]
[2015-05-27 19:58:41,274][DEBUG][cluster.service          ] [Trick Shot] publishing cluster state version 5
[2015-05-27 19:58:41,274][DEBUG][cluster.service          ] [Trick Shot] set local cluster state to version 5
[2015-05-27 19:58:41,277][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: execute
[2015-05-27 19:58:41,293][DEBUG][cluster.action.index     ] [Trick Shot] successfully updated master with mapping update: index [_river], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], type [nc-search] and source [{"nc-search":{"properties":{"bulk_flush_interval":{"type":"string"},"bulk_size":{"type":"long"},"interval":{"type":"string"},"keep_expunged_messages":{"type":"boolean"},"ldap_name_field":{"type":"string"},"mail.debug":{"type":"string"},"mail.imap.connectionpoolsize":{"type":"string"},"mail.imap.host":{"type":"string"},"mail.imap.port":{"type":"long"},"mail.imap.ssl.enable":{"type":"boolean"},"mail.imap.timeout":{"type":"long"},"mail.store.protocol":{"type":"string"},"mail_index_name":{"type":"string"},"mail_type_name":{"type":"string"},"max_bulk_requests":{"type":"string"},"password":{"type":"string"},"threads":{"type":"long"},"type":{"type":"string"},"user":{"type":"string"},"with_attachments":{"type":"boolean"},"with_flag_sync":{"type":"boolean"},"with_striptags_from_textcontent":{"type":"boolean"},"with_text_content":{"type":"boolean"}}}}]
[2015-05-27 19:58:41,293][DEBUG][cluster.service          ] [Trick Shot] processing [update-mapping [_river][nc-search] / node [5izIMXpCQVWKMV4L9gI9rw], order [1]]: done applying updated cluster_state (version: 5)
[2015-05-27 19:58:41,325][DEBUG][river.routing            ] [Trick Shot] no river _meta document found retrying in 1000 ms
[2015-05-27 19:58:41,355][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-05-27 19:58:42,208][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: execute
[2015-05-27 19:58:42,208][DEBUG][river.routing            ] [Trick Shot] _river/nc-search/_meta document found.
[2015-05-27 19:58:42,223][DEBUG][river.routing            ] [Trick Shot] going to allocate river [nc-search] on node [Trick Shot][5izIMXpCQVWKMV4L9gI9rw][VM-2008R2][inet[/192.168.1.6:9300]]
[2015-05-27 19:58:42,223][DEBUG][river.cluster            ] [Trick Shot] cluster state updated, version [1], source [reroute_rivers_node_changed]
[2015-05-27 19:58:42,223][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: done applying updated cluster_state
[2015-05-27 19:58:42,223][DEBUG][river                    ] [Trick Shot] creating river [imap][nc-search]
[2015-05-27 19:58:42,298][DEBUG][de.saly.elasticsearch.river.imap.IMAPRiver] river settings {master_user=null, ldap_user=null, mail.debug=true, user_source=null, mail.imap.timeout=10000, mail.imap.host=, index_settings=null, bulk_size=100, type=imap, max_bulk_requests=30, ldap_password=null, with_attachments=false, password=, with_striptags_from_textcontent=true, ldap_name_field=uid, mail.imap.ssl.enable=false, ldap_url=null, with_text_content=true, folderpattern=null, with_flag_sync=true, threads=5, mail_index_name=imapriverdata, mail.imap.connectionpoolsize=3, mail.imap.port=993, bulk_flush_interval=5s, schedule=null, keep_expunged_messages=false, ldap_base=null, ldap_password_field=null, ldap_refresh_interval=null, master_password=null, mail_type_name=mail, mail.store.protocol=imap, interval=60s, user=, type_mapping=null}
[2015-05-27 19:58:42,299][DEBUG][de.saly.elasticsearch.river.imap.IMAPRiver] mail settings {mail.imap.port=993, mail.imap.connectionpoolsize=3, mail.debug=true, mail.imap.host=, mail.imap.ssl.enable=false, mail.imap.timeout=10000, mail.store.protocol=imap}
[2015-05-27 19:58:42,414][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: execute
[2015-05-27 19:58:42,417][DEBUG][river.cluster            ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state
[2015-05-27 19:58:42,481][INFO ][de.saly.elasticsearch.river.imap.IMAPRiver] IMAPRiver created, river name: nc-search
[2015-05-27 19:58:42,481][INFO ][de.saly.elasticsearch.river.imap.IMAPRiver] Start IMAPRiver ...
[2015-05-27 19:58:42,481][DEBUG][de.saly.elasticsearch.maildestination.ElasticsearchBulkMailDestination] waiting for cluster state YELLOW
salyh commented 9 years ago

Which river version and which jdk (vendor and version) do you have in place?

Von meinem iPhone gesendet

Am 27.05.2015 um 14:29 schrieb Dean Smith notifications@github.com:

I found some of my logs. This is from ES 1.4.5, where I think River-IMAP was also causing ES to hang when starting up to poll an inbox:

[2015-05-27 19:55:41,608][DEBUG][plugins ] [Trick Shot] [C:\apps\elasticsearch\plugins\river-imap_site] directory does not exist. [2015-05-27 19:58:21,131][DEBUG][plugins ] [Trick Shot] [C:\apps\elasticsearch\plugins\river-imap_site] directory does not exist.

[2015-05-27 19:58:39,993][DEBUG][indices ] [Trick Shot] creating Index [_river], shards [1]/[1] [2015-05-27 19:58:40,490][DEBUG][index.mapper ] [Trick Shot] [_river] using dynamic[true], default mapping: default_mapping_location[null], loaded_from[jar:file:/C:/apps/elasticsearch/lib/elasticsearch-1.4.5.jar!/org/elasticsearch/index/mapper/default-mapping.json], default percolator mapping: location[null], loaded_from[null] [2015-05-27 19:58:40,490][DEBUG][index.cache.query.parser.resident] [Trick Shot] [_river] using [resident] query cache with max_size [100], expire [null] [2015-05-27 19:58:40,490][DEBUG][index.store.fs ] [Trick Shot] [_river] using index.store.throttle.type [node], with index.store.throttle.max_bytes_per_sec [0b] [2015-05-27 19:58:40,671][INFO ][cluster.metadata ] [Trick Shot] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings [nc-search] [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing ... (reason [cleaning up after validating index on master]) [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing index service [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing index cache [2015-05-27 19:58:40,686][DEBUG][index.cache.filter.weighted] [Trick Shot] [_river] full cache clear, reason [close] [2015-05-27 19:58:40,686][DEBUG][index.cache.fixedbitset ] [Trick Shot] [_river] Clearing all FixedBitSets because [close] [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] clearing index field data [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing analysis service [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing index engine [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing index gateway [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing mapper service [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing index query parser service [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closing index service [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] [_river] closed... (reason [cleaning up after validating index on master]) [2015-05-27 19:58:40,686][DEBUG][cluster.service ] [Trick Shot] cluster state updated, version [3], source [create-index [_river], cause [auto(index api)]] [2015-05-27 19:58:40,686][DEBUG][cluster.service ] [Trick Shot] publishing cluster state version 3 [2015-05-27 19:58:40,686][DEBUG][cluster.service ] [Trick Shot] set local cluster state to version 3 [2015-05-27 19:58:40,686][DEBUG][indices.cluster ] [Trick Shot] [_river] creating index [2015-05-27 19:58:40,686][DEBUG][indices ] [Trick Shot] creating Index [_river], shards [1]/[1] [2015-05-27 19:58:40,746][DEBUG][index.mapper ] [Trick Shot] [_river] using dynamic[true], default mapping: default_mapping_location[null], loaded_from[jar:file:/C:/apps/elasticsearch/lib/elasticsearch-1.4.5.jar!/org/elasticsearch/index/mapper/default-mapping.json], default percolator mapping: location[null], loaded_from[null] [2015-05-27 19:58:40,746][DEBUG][index.cache.query.parser.resident] [Trick Shot] [_river] using [resident] query cache with max_size [100], expire [null] [2015-05-27 19:58:40,746][DEBUG][index.store.fs ] [Trick Shot] [_river] using index.store.throttle.type [node], with index.store.throttle.max_bytes_per_sec [0b] [2015-05-27 19:58:40,746][DEBUG][indices.cluster ] [Trick Shot] [_river] adding mapping [nc-search], source [{"nc-search":{"properties":{}}}] [2015-05-27 19:58:40,746][DEBUG][indices.cluster ] [Trick Shot] [_river][0] creating shard [2015-05-27 19:58:40,746][DEBUG][index.service ] [Trick Shot] [_river] creating shard_id [0] [2015-05-27 19:58:40,972][DEBUG][index.store ] [Trick Shot] [_river][0] store stats are refreshed with refresh_interval [10s] [2015-05-27 19:58:40,972][DEBUG][index.deletionpolicy ] [Trick Shot] [_river][0] Using [keep_only_last] deletion policy [2015-05-27 19:58:40,972][DEBUG][index.merge.policy ] [Trick Shot] [_river][0] using [tiered] merge mergePolicy with expunge_deletes_allowed[10.0], floor_segment[2mb], max_merge_at_once[10], max_merge_at_once_explicit[30], max_merged_segment[5gb], segments_per_tier[10.0], reclaim_deletes_weight[2.0] [2015-05-27 19:58:40,972][DEBUG][index.merge.scheduler ] [Trick Shot] [_river][0] using [concurrent] merge scheduler with max_thread_count[1], max_merge_count[3] [2015-05-27 19:58:40,972][DEBUG][index.shard.service ] [Trick Shot] [_river][0] state: [CREATED] [2015-05-27 19:58:40,972][DEBUG][index.shard.service ] [Trick Shot] [_river][0] scheduling optimizer / merger every 1s [2015-05-27 19:58:40,987][DEBUG][index.translog ] [Trick Shot] [_river][0] interval [5s], flush_threshold_ops [2147483647], flush_threshold_size [200mb], flush_threshold_period [30m] [2015-05-27 19:58:40,987][DEBUG][index.shard.service ] [Trick Shot] [_river][0] state: [CREATED]->[RECOVERING], reason [from gateway] [2015-05-27 19:58:40,987][DEBUG][index.gateway ] [Trick Shot] [_river][0] starting recovery from local ... [2015-05-27 19:58:41,008][DEBUG][cluster.service ] [Trick Shot] processing [create-index [_river], cause [auto(index api)]]: done applying updated cluster_state (version: 3)

[2015-05-27 19:58:41,036][DEBUG][index.engine.internal ] [Trick Shot] [_river][0] starting engine [2015-05-27 19:58:41,051][DEBUG][river.cluster ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state [2015-05-27 19:58:41,136][DEBUG][index.shard.service ] [Trick Shot] [_river][0] scheduling refresher every 1s [2015-05-27 19:58:41,137][DEBUG][index.shard.service ] [Trick Shot] [_river][0] state: [RECOVERING]->[POST_RECOVERY], reason [post recovery from gateway, no translog] [2015-05-27 19:58:41,137][DEBUG][index.gateway ] [Trick Shot] [_river][0] recovery completed from [local], took [150ms] [2015-05-27 19:58:41,137][DEBUG][cluster.action.shard ] [Trick Shot] sending shard started for [_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], reason [after recovery from gateway] [2015-05-27 19:58:41,138][DEBUG][cluster.action.shard ] [Trick Shot] received shard started for [_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], reason [after recovery from gateway]

[2015-05-27 19:58:41,139][DEBUG][cluster.action.shard ] [Trick Shot] [_river][0] will apply shard started [_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], reason [after recovery from gateway] [2015-05-27 19:58:41,141][DEBUG][cluster.service ] [Trick Shot] cluster state updated, version [4], source [shard-started ([_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING]), reason [after recovery from gateway]] [2015-05-27 19:58:41,141][DEBUG][cluster.service ] [Trick Shot] publishing cluster state version 4 [2015-05-27 19:58:41,141][DEBUG][cluster.service ] [Trick Shot] set local cluster state to version 4 [2015-05-27 19:58:41,143][DEBUG][index.shard.service ] [Trick Shot] [_river][0] state: [POST_RECOVERY]->[STARTED], reason [global state is [STARTED]]

[2015-05-27 19:58:41,148][DEBUG][cluster.service ] [Trick Shot] processing [shard-started ([_river][0], node[5izIMXpCQVWKMV4L9gI9rw], [P], s[INITIALIZING]), reason [after recovery from gateway]]: done applying updated cluster_state (version: 4) [2015-05-27 19:58:41,164][DEBUG][river.routing ] [Trick Shot] no river _meta document found retrying in 1000 ms [2015-05-27 19:58:41,164][DEBUG][river.cluster ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state

[2015-05-27 19:58:41,210][DEBUG][cluster.metadata ] [Trick Shot] [_river] update_mapping nc-search with source [{"nc-search":{"properties":{"bulk_flush_interval":{"type":"string"},"bulk_size":{"type":"long"},"interval":{"type":"string"},"keep_expunged_messages":{"type":"boolean"},"ldap_name_field":{"type":"string"},"mail.debug":{"type":"string"},"mail.imap.connectionpoolsize":{"type":"string"},"mail.imap.host":{"type":"string"},"mail.imap.port":{"type":"long"},"mail.imap.ssl.enable":{"type":"boolean"},"mail.imap.timeout":{"type":"long"},"mail.store.protocol":{"type":"string"},"mail_index_name":{"type":"string"},"mail_type_name":{"type":"string"},"max_bulk_requests":{"type":"string"},"password":{"type":"string"},"threads":{"type":"long"},"type":{"type":"string"},"user":{"type":"string"},"with_attachments":{"type":"boolean"},"with_flag_sync":{"type":"boolean"},"with_striptags_from_textcontent":{"type":"boolean"},"with_text_content":{"type":"boolean"}}}}] [2015-05-27 19:58:41,268][DEBUG][cluster.service ] [Trick Shot] cluster state updated, version [5], source [update-mapping [_river][nc-search] / node [5izIMXpCQVWKMV4L9gI9rw], order [1]] [2015-05-27 19:58:41,274][DEBUG][cluster.service ] [Trick Shot] publishing cluster state version 5 [2015-05-27 19:58:41,274][DEBUG][cluster.service ] [Trick Shot] set local cluster state to version 5

[2015-05-27 19:58:41,293][DEBUG][cluster.action.index ] [Trick Shot] successfully updated master with mapping update: index [_river], indexUUID [9dP8Ow_rQtSWtR498Y7o-g], type [nc-search] and source [{"nc-search":{"properties":{"bulk_flush_interval":{"type":"string"},"bulk_size":{"type":"long"},"interval":{"type":"string"},"keep_expunged_messages":{"type":"boolean"},"ldap_name_field":{"type":"string"},"mail.debug":{"type":"string"},"mail.imap.connectionpoolsize":{"type":"string"},"mail.imap.host":{"type":"string"},"mail.imap.port":{"type":"long"},"mail.imap.ssl.enable":{"type":"boolean"},"mail.imap.timeout":{"type":"long"},"mail.store.protocol":{"type":"string"},"mail_index_name":{"type":"string"},"mail_type_name":{"type":"string"},"max_bulk_requests":{"type":"string"},"password":{"type":"string"},"threads":{"type":"long"},"type":{"type":"string"},"user":{"type":"string"},"with_attachments":{"type":"boolean"},"with_flag_sync":{"type":"boolean"},"with_striptags_from_textcontent":{" type":"boolean"},"with_text_content":{"type":"boolean"}}}}] [2015-05-27 19:58:41,293][DEBUG][cluster.service ] [Trick Shot] processing [update-mapping [_river][nc-search] / node [5izIMXpCQVWKMV4L9gI9rw], order [1]]: done applying updated cluster_state (version: 5) [2015-05-27 19:58:41,325][DEBUG][river.routing ] [Trick Shot] no river _meta document found retrying in 1000 ms [2015-05-27 19:58:41,355][DEBUG][river.cluster ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state

[2015-05-27 19:58:42,208][DEBUG][river.routing ] [Trick Shot] _river/nc-search/_meta document found. [2015-05-27 19:58:42,223][DEBUG][river.routing ] [Trick Shot] going to allocate river [nc-search] on node [Trick Shot][5izIMXpCQVWKMV4L9gI9rw][VM-2008R2][inet[/192.168.1.6:9300]] [2015-05-27 19:58:42,223][DEBUG][river.cluster ] [Trick Shot] cluster state updated, version [1], source [reroute_rivers_node_changed] [2015-05-27 19:58:42,223][DEBUG][river.cluster ] [Trick Shot] processing [reroute_rivers_node_changed]: done applying updated cluster_state [2015-05-27 19:58:42,223][DEBUG][river ] [Trick Shot] creating river [imap][nc-search] [2015-05-27 19:58:42,298][DEBUG][de.saly.elasticsearch.river.imap.IMAPRiver] river settings {master_user=null, ldap_user=null, mail.debug=true, user_source=null, mail.imap.timeout=10000, mail.imap.host=, index_settings=null, bulk_size=100, type=imap, max_bulk_requests=30, ldap_password=null, with_attachments=false, password=, with_striptags_from_textcontent=true, ldap_name_field=uid, mail.imap.ssl.enable=false, ldap_url=null, with_text_content=true, folderpattern=null, with_flag_sync=true, threads=5, mail_index_name=imapriverdata, mail.imap.connectionpoolsize=3, mail.imap.port=993, bulk_flush_interval=5s, schedule=null, keep_expunged_messages=false, ldap_base=null, ldap_password_field=null, ldap_refresh_interval=null, master_password=null, mail_type_name=mail, mail.store.protocol=imap, interval=60s, user=, type_mapping=null} [2015-05-27 19:58:42,299][DEBUG][de.saly.elasticsearch.river.imap.IMAPRiver] mail settings {mail.imap.port=993, mail.imap.connectionpoolsize=3, mail.debug=true, mail.imap.host=, mail.imap.ssl.enable=false, mail.imap.timeout=10000, mail.store.protocol=imap}

[2015-05-27 19:58:42,417][DEBUG][river.cluster ] [Trick Shot] processing [reroute_rivers_node_changed]: no change in cluster_state [2015-05-27 19:58:42,481][INFO ][de.saly.elasticsearch.river.imap.IMAPRiver] IMAPRiver created, river name: nc-search [2015-05-27 19:58:42,481][INFO ][de.saly.elasticsearch.river.imap.IMAPRiver] Start IMAPRiver ... [2015-05-27 19:58:42,481][DEBUG][de.saly.elasticsearch.maildestination.ElasticsearchBulkMailDestination] waiting for cluster state YELLOW — Reply to this email directly or view it on GitHub.

deancsmith commented 9 years ago

River version 0.4 (from here, installed as per the Readme).

I'm using the Oracle JDK 1.8.0_45.

salyh commented 9 years ago

ah, bug #18 is fixed after 0.4. pls. use current master and build it on your own? or i can send you a build if youre not a developer.

On Wed, May 27, 2015 at 2:51 PM, Dean Smith notifications@github.com wrote:

River version 0.4 (from here http://dl.bintray.com/salyh/maven/de/saly/elasticsearch/plugin/elasticsearch-river-imap/0.4/elasticsearch-river-imap-0.4-plugin.zip, installed as per the Readme).

I'm using the Oracle JDK 1.8.0_45.

— Reply to this email directly or view it on GitHub https://github.com/salyh/elasticsearch-river-imap/issues/18#issuecomment-105894873 .

Hendrik Saly (salyh, hendrikdev22) @hendrikdev22 PGP: 0x22D7F6EC

deancsmith commented 9 years ago

Ah, my apologies, I must have mis-read the commit dates above.

I haven't built a ES plugin before. I'll have a go and let you know if I have any problems. Thanks!

salyh commented 9 years ago

ok, thx

On Wed, May 27, 2015 at 4:36 PM, Dean Smith notifications@github.com wrote:

Ah, my apologies, I must have mis-read the commit dates above.

I haven't built a ES plugin before. I'll have a go and let you know if I have any problems. Thanks!

— Reply to this email directly or view it on GitHub https://github.com/salyh/elasticsearch-river-imap/issues/18#issuecomment-105934400 .

Hendrik Saly (salyh, hendrikdev22) @hendrikdev22 PGP: 0x22D7F6EC

salyh commented 9 years ago

@deancsmith 0.8.5 is released which contains the fix for this issue. just download the plugin from here https://github.com/salyh/elasticsearch-river-imap/releases/tag/v0.8.5 or install it via 

bin/plugin -i de.saly/elasticsearch-river-imap/0.8.5

deancsmith commented 9 years ago

Thanks @salyh, your version 0.8.5 works perfectly with ES 1.5!