search

dieswaytoofast / erlasticsearch

Erlang thrift interface to elastic_search
45 stars 18 forks source link

Core dump if elasticsearch is down #17

Open jeromer opened 10 years ago

jeromer commented 10 years ago

I am not sure if this is intended or not but if we send a document and elasticsearch is down poolboy seems to try to reconnect again and again until the entire application (not only erlasticsearch) terminates and core dumps.

I tried to define retry_interval and retry_amount to super high values for testing but it changed nothing : 

        {connection_options, [
            {thrift_host, "localhost"},
            {thrift_port, 9500},
            {thrift_options, [
                {framed, false}]
            },
            {binary_response, true},
            {retry_interval, 666666},
            {retry_amount, 666666}
        ]}

Here is an excerpt of the errors I get

[...]
23:10:44.042 [error] CRASH REPORT Process <0.4161.0> with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}} in erlasticsearch_poolboy_worker:init/1 line 40 in gen_server:init_it/6 line 328
23:10:44.042 [error] Supervisor {<0.4160.0>,poolboy_sup} had child erlasticsearch_poolboy_worker started with {erlasticsearch_poolboy_worker,start_link,undefined} at <0.4161.0> exit with reason no match of right hand value {error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}} in erlasticsearch_poolboy_worker:init/1 line 40 in context child_terminated
23:10:44.042 [error] CRASH REPORT Process <0.4159.0> with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_li..."},...]}]}}},...}} in poolboy:new_worker/1 line 232 in gen_server:init_it/6 line 328
23:10:44.044 [error] Supervisor erlasticsearch_poolboy_sup had child 'erlasticsearch___access_logs.pool' started with poolboy:start_link([{name,{local,'erlasticsearch___access_logs.pool'}},{worker_module,erlasticsearch_poolboy_worker},...], [{thrift_host,"localhost"},{thrift_port,9500},{thrift_options,[{framed,false}]},{binary_response,...},...]) at undefined exit with reason no match of right hand value {error,{{badmatch,{error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_li..."},...]}]}}},...}} in poolboy:new_worker/1 line 232 in context start_error
23:10:44.044 [error] gen_server <0.4160.0> terminated with reason: no match of right hand value {error,{{badmatch,{error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_li..."},...]}]}}},...}} in poolboy:new_worker/1 line 232
23:10:44.044 [error] CRASH REPORT Process <0.4160.0> with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_li..."},...]}]}}},...}} in poolboy:new_worker/1 line 232 in gen_server:terminate/6 line 744
23:10:44.045 [error] Supervisor erlasticsearch_sup had child erlasticsearch_poolboy_sup started with erlasticsearch_poolboy_sup:start_link() at {restarting,<0.632.0>} exit with reason {shutdown,{failed_to_start_child,'erlasticsearch___access_logs.pool',{{badmatch,{error,{{badmatch,{error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[...]},...]}}},...}}},...}}} in context start_error
23:10:44.047 [error] CRASH REPORT Process <0.4168.0> with 0 neighbours exited with reason: no match of right hand value {error,econnrefused} in thrift_binary_protocol:'-new_protocol_factory/2-fun-0-'/2 line 336 in gen_server:init_it/6 line 328
23:10:44.073 [error] CRASH REPORT Process <0.4167.0> with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,econnrefused}},[{thrift_binary_protocol,'-new_protocol_factory/2-fun-0-',2,[{file,"src/thrift_binary_protocol.erl"},{line,336}]},{thrift_client_util,new,4,[{file,"src/thrift_client_util.erl"},{line,59}]},{erlasticsearch,connection,1,[{file,"src/erlasticsearch.erl"},{line,839}]},{erlasticsearch,init,1,[{file,"src/erlasticsearch.erl"},{line,609}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}} in erlasticsearch_poolboy_worker:init/1 line 40 in gen_server:init_it/6 line 328
[...]
dieswaytoofast commented 10 years ago

Yeah, its a "feature :-( (A partial) fix for this is to plugin a backoff mechanism...