Any documents about reason_term?

[jackalcooper]

I'm using Maxwell to build load testing tool. And I got reason term connection_closed or retry_later in the return of a post or get method. {:error, reason_term, %Maxwell.Conn{}}. But I can't find anything about connection_closed or retry_later in the doc or the source code.

[secretworry]

@jackalcooper The short answer is they are returned by ibrowse adapter. Basically,

• the error retry_later get returned when you are trying to pipe too many connections to the same host at the same time. The default maximum of concurrent connections is 10, you can modify the option named max_pipeline_size to alter this for a larger throughput.
• the error connection_closed get returned when you are trying to send requests to a socket that is closing (usually by remote).

[jackalcooper]

Thanks. I've found the document

[jackalcooper]

Hi, @secretworry @zhongwencool I kept getting this crash. Does it have anything to do the fact that I kept passing different Adapter options? Or it's because max_sessions I set is too small?

crasher:
    initial call: Elixir.Tuna.User.IOSUser:init/1
    pid: <0.3538.0>
    registered_name: []
    exception exit: {timeout,
                        {gen_server,call,
                            [<0.3634.0>,
                             {spawn_connection,
                                 {url,
                                     "http://test3.devserver.com:8033/api/v1/conversations/from_last_seen",
                                     "test3.devserver.com",8033,undefined,
                                     undefined,
                                     "/api/v1/conversations/from_last_seen",
                                     http,hostname},
                                 10,1000000,
                                 {[],false},
                                 []}]}}
      in function  gen_server:call/2 (gen_server.erl, line 204)
      in call from ibrowse:try_routing_request/14 (/root/subway_umbrella/deps/ibrowse/src/ibrowse.erl, line 377)
      in call from 'Elixir.Maxwell.Adapter.Ibrowse':send_direct/1 (lib/maxwell/adapter/ibrowse.ex, line 14)
      in call from 'Elixir.Maxwell.Adapter.Ibrowse':call/1 (lib/maxwell/adapter/adapter.ex, line 28)
      in call from 'Elixir.Maxwell.Middleware.Json':call/3 (lib/maxwell/middleware/middleware.ex, line 29)
      in call from 'Elixir.Maxwell.Middleware.Opts':call/3 (lib/maxwell/middleware/middleware.ex, line 29)
      in call from 'Elixir.Maxwell.Middleware.Headers':call/3 (lib/maxwell/middleware/middleware.ex, line 29)
      in call from 'Elixir.Maxwell.Middleware.BaseUrl':call/3 (lib/maxwell/middleware/middleware.ex, line 29)
    ancestors: [<0.2121.0>,<0.1811.0>]
    messages: [{'$gen_cast',
                      {offline,
                          [{connect_timeout,200},
                           {recv_timeout,200},
                           {max_pipeline_size,50},
                           {max_sessions,10}]}},
                  {'$gen_cast',
                      {online,
                          [{connect_timeout,200},
                           {recv_timeout,200},
                           {max_pipeline_size,50},
                           {max_sessions,10}]}},
                  {'$gen_cast',
                      {post_login,
                          [{connect_timeout,200},
                           {recv_timeout,200},
                           {max_pipeline_size,50},
                           {max_sessions,10}]}},
                  {'$gen_cast',
                      {offline,
                          [{connect_timeout,200},
                           {recv_timeout,200},
                           {max_pipeline_size,50},
                           {max_sessions,10}]}}]

[jackalcooper]

I read that ibrowse's options is per {host, port}. Is there an extra way to associate them?

[zhongwencool]

1. Your have config 10 session, every session can handle 50 request. So almost 500 request at the same time.

2. But timeout due to ibrowse_lb(every {host, port} manage by one ibrowse_lb, you ibrowse_lb is too busy to manage requests.

3. ibrowse_lb do very simple thing, I can not figure out why It's busy.

4. Two ways to figure out:

   :ibrowse.show_dest_status to get LB pid :erlang.process_info(:erlang.list_to_pid('<x.x.x>')) to see status, message_queue_len, messages

OR just using observer GUI to see ibrowse_lb status :observer.start()

message_queue_len!

http://www.cnblogs.com/zhongwencool/p/ibrowse.html This maybe help to understand ibrowse. :)

[jackalcooper]

@zhongwencool thanks for your replying. So what is going to happen to those requests in queue if I change the max sessions to a smaller number in a Maxwell function call? I give it a new options keyword list every time. Maybe it has anything to do with the crash?

[zhongwencool]

@jackalcooper I think I know why request timeout...

{timeout,
                        {gen_server,call,
                            [<0.3634.0>,
                             {spawn_connection,
                                 {url,
                                     "http://test3.devserver.com:8033/api/v1/conversations/from_last_seen",
                                     "test3.devserver.com",8033,undefined,
                                     undefined,
                                     "/api/v1/conversations/from_last_seen",
                                     http,hostname},
                                 10,1000000,
                                 {[],false},
                                 []}]}}

{spawn_connection, Url, Max_sess, Max_pipe, SSL_options, Process_options} You have setting Max_sess=10 and Max_pipe =1000,000 ...... It's mean every worker should hold 1000,000 request at the same time. Maybe setting more session will better.

[secretworry]

@jackalcooper I think the problem here is that using a client with pooled connetions to do bechmark test is to test the pool implmentation of the client more than to test your server. The reason of using a pool is to restrict the resource usage(expect to send limited requests) Two alternatives are here:

• Eliminating the pool by either increasing restrictions(max session, max pipe) or using an adpater without pool.
• Increasing the number of concurrent connections smoothly(don't send the next request before receive a response). So the requests won't crash the client before actually reach your server.

For your case, I recommend you

• to increase the Max_sess which is roughly equal to concurrency, and decrease the Max_pipe, which may lead to a longer response time( with additional queue timei)
• handle retry_later error in your program to prevent choking the connection pool.

[jackalcooper]

@zhongwencool @secretworry Thanks guys. I think I went to the wrong direction. My boss wants me to implement the benchmark for each test case at fixed session count(netstat -an|grep :80|grep EST|wc -l). And I have to do it by launching one instance of BEAM for each test case and set a very low max_session. At first I set a huge max_session and my boss thought it makes little sense. I guess I have to do some changes to my user simulation mechanism at a high max_session config.

[secretworry]

@jackalcooper I would recommend you to use ab, and collect its output.

[jackalcooper]

@secretworry May I ask what ab do you mean?

[secretworry]

@jackalcooper ab is for Apache HTTP server benchmarking tool, a command line tool. If all you need is to measure request time, ab will definitely give you a better result.

[jackalcooper]

@secretworry never heard of it. I'm using gen state machine to simulate user and store the state between requests. I chose to build my own rather than using the existed because I want to have the ability to save state like token or different ids for querying other things.

[jackalcooper]

I found that Hackney support specific connection pool which suits my need.