Couchbeam blocks after a while

[gdamjan]

Couchbeam as of 2dd34263 blocks when I send it this barrage of requests, from the couchdb logs I see 150 requests were made. The database didn't exist so the responses were 404. CouchDB otherwise still works fine (1.6.x).

The example code:

-module(test).

-export([main/0]).

main() ->
    couchbeam:start(),
    Server = couchbeam:server_connection(<<"http://localhost:5984">>, []),
    {ok, Db} = couchbeam:open_db(Server, <<"test">>, []),
    a_lot_of_puts(Db, 100000).

a_lot_of_puts(_, 0) ->
    ok;

a_lot_of_puts(Db, N) ->
    M = list_to_binary(integer_to_list(N)),
    log_message(Db, <<"tester">>, <<"testing">>, <<"message ", M/binary>>),
    a_lot_of_puts(Db, N-1).

log_message(Db, Sender, Channel, Message) ->
    {MegaSecs, Secs, MicroSecs} = now(),
    Timestamp = MegaSecs * 1000000 + Secs + MicroSecs/1000000,
    Doc = {[
            {<<"sender">>, Sender},
            {<<"channel">>, Channel},
            {<<"message">>, Message},
            {<<"timestamp">>, Timestamp}
           ]},
    catch couchbeam:save_doc(Db, Doc).

retrying it now when the database does exists.

[gdamjan]

I don't know if I'm debugging this right, but the problem seems to happen in

** exception exit: {killed,{gen_server,call,
                                   [<0.64.0>,
                                    {checkout,{"localhost",5984,hackney_tcp_transport},
                                              <0.39.0>,#Ref<0.0.0.3021>},
                                    infinity]}}
     in function  gen_server:call/3 (gen_server.erl, line 190)
     in call from hackney_pool:checkout/4 (src/hackney_connect/hackney_pool.erl, line 65)

[gdamjan]

I replaced infinity there with 10000 and now it throws an exception:

** exception exit: {timeout,{gen_server,call,
                                    [<0.64.0>,
                                     {checkout,{"localhost",5984,hackney_tcp_transport},
                                               <0.39.0>,#Ref<0.0.0.2202>},
                                     10000]}}
 in function  gen_server:call/3 (gen_server.erl, line 190)
 in call from hackney_pool:checkout/4 (src/hackney_connect/hackney_pool.erl, line 65)
 in call from hackney_connect:socket_from_pool/4 (src/hackney_connect/hackney_connect.erl, line 173)
 in call from hackney_connect:connect/5 (src/hackney_connect/hackney_connect.erl, line 30)
 in call from hackney:request/5 (src/hackney_client/hackney.erl, line 309)
 in call from couchbeam_httpc:db_request/6 (src/couchbeam_httpc.erl, line 25)
 in call from couchbeam:save_doc/4 (src/couchbeam.erl, line 586)
 in call from test:a_lot_of_puts/2 (test.erl, line 16)

[gdamjan]

ok this bug needs to go to hackney_pool. I tried master too.

the checkout handler returns {noreply, State2#state{queues = Queues2, nb_waiters=NbWaiters2}}; so I guess the call is never replied to.

[epappas]

Hello there,

Please correct me if I'm wrong, I just run your test; couchbeam from master (1.1.4), couchDB 1.6.1, Erlang/OTP 17

And couchbeam:save_doc(Db, Doc) is giving me a {error,not_found}, no crashing, no halting (and no docs created obviously).

Any ideas?

Cheers.

[gdamjan]

@epappas sure, when doing a single save_doc it'll return {error,not_found} since couchdb returns 404. the problem is after some 150 iterations in quick succession of this call, it seems the hackney_pool stops giving out connections.

[gdamjan]

What's interesting is that successful save_docs / PUTs don't have this problem (at least up to some 6000 saves). Just a wild guess that couchdb or hackney behaves differently on 404 responses.

Testing… I see that after the 404 responses, there's no http connection in netstat, and couchbeam will use a new connection. With the 201 responses, the http connection is kept alive and successive requests go over the same connection.

[benoitc]

Seems like couchbeam is blocked waiting the attempts to connect to couchdb timeout. Adding the option {connect_timeout, 200} (ie. a timeout of 200ms) to the db solved it there:

{ok, Db} = couchbeam:open_db(Server, <<"test">>, [{connect_timeout, 200}]).

Can you confirm?

[gdamjan]

Nope, same thing happens. On the 404 errors CouchDB closes the connection, it seems that hackney_pool doesn't handle that so well (but I still don't understand how that works).

[benoitc]

thanks for the feedback. It helped a lot :)