search

william-os4y / fapws3

Fast Asynchronous Python Web Server (based on libev)
GNU General Public License v2.0
341 stars 38 forks source link

FAPWS3 crashed behind Haproxy #36

Closed wigunawan closed 7 years ago

wigunawan commented 12 years ago

I found that fapws3 crashed behind haproxy load balancer. here is the screenshot url: http://img521.imageshack.us/img521/2727/fapws3behindhaproxycras.png

fapws3 crashed immediately after i start haproxy daemon, or after i stop/start fapws instance. my fapws3 application is just a hello world app. i think it crashed when haproxy do a health check, maybe.

here is my haproxy config, thanks:

global log 127.0.0.1 local0 log 127.0.0.1 local1 notice

log loghost local0 info

    maxconn 4096
    #debug
    #quiet
    user haproxy
    group haproxy

defaults log global mode http option httplog option dontlognull retries 3 option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000

backend rpc_rmq_01 mode http balance roundrobin server rmq_01_01 127.0.0.1:8997 check server rmq_01_02 127.0.0.1:8999 check

backend default_rpc mode http balance roundrobin server mainwebsite 127.0.0.1:80 check

frontend myfrontend *:8998 log global maxconn 25000 option forwardfor acl acl_rmq_01 url_beg /rpc/rmq-01 use_backend rpc_rmq_01 if acl_rmq_01 default_backend default_rpc

william-os4y commented 12 years ago

I cannot reproduce the problem :-(. I've install haproxy and copied your config.

Then I've started 2 Fapws servers one on 8997 and one on 8999

Then I've browsed to 127.0.0.1:8998 with 2 different url:

And I've received the following message: [wi@myarchhost hello]$ python2 test.py 8997 listen on 0.0.0.0:8997 Using epoll as event backend uri not found:/rpc/rmq-01

AND

[wi@myarchhost hello]$ python2 test.py 8999 listen on 0.0.0.0:8999 Using epoll as event backend uri not found:/rpc/rmq-01/hello

We see that Haproxy has request both Fapws servers (roundrobin effect I suppose).

None of Fapws servers have crashed.

Could you precise the scenario to allow me to reproduce the problem ?

I'm running on : Linux myarchhost 3.3.1-1-ARCH #1 SMP PREEMPT with python-2.7

wigunawan commented 12 years ago

Thanks for your response William, i've upload the video to show you more, on this video segmentation fault occured.

http://youtu.be/pF-5erF1_Kk

after i stop haproxy, then start test.py again, the error was gone. but when i start haproxy, the error occured again immediately, even without requesting any url.

caruccio commented 12 years ago

hi, It's easier with a propper backtrace. Try this:

ulimit -c unlimited

python test.py 8999

This should generate a file named "core" (or "core.XXX" where XXX is the pid of the process). Send this to someone or make it available to download.

EDIT: sorry, didn't saw it was already fixed.

wigunawan commented 12 years ago

caruccio, here is the file you expected: http://www.4shared.com/rar/qd334heQ/core.html

the problem has not been solved.

wigunawan commented 12 years ago

i found it more strange, the problem didn't occured when i add httpchk option in haproxy configuration. i don't know why... what's going on exactly?

backend rpc_rmq_01 mode http balance roundrobin option httpchk HEAD /rpc/rmq-01/healthcheck HTTP/1.0 server rmq_01_01 127.0.0.1:8997 check server rmq_01_02 127.0.0.1:8999 check

william-os4y commented 12 years ago

I've carefully look at your video ... but I cannot reproduce the problem on my machine :-(.

Could I ask you to use the last Fapws version (master) ? I've just, for an another issue, made some changes. Your problem is maybe linked to this.

Concerning the core file ... I cannot download it (require signup, ...).

But if you could copy./paste the result of the backtrace, this would be help full.

Thanks.

On Wed, Apr 11, 2012 at 2:01 PM, wigunawan reply@reply.github.com wrote:

i found it more strange, the problem didn't occured when i add httpchk option in haproxy configuration. i don't know why... what's going on exactly?

backend rpc_rmq_01        mode http        balance roundrobin        option httpchk HEAD /rpc/rmq-01/healthcheck HTTP/1.0        server rmq_01_01 127.0.0.1:8997 check        server rmq_01_02 127.0.0.1:8999 check

Reply to this email directly or view it on GitHub: https://github.com/william-os4y/fapws3/issues/36#issuecomment-5067145

wigunawan commented 12 years ago

William, please try this link: http://www.ziddu.com/download/19126420/core.rar.html

wigunawan commented 12 years ago

William, after i set debug mode in fapws3, i see that haproxy send following request periodically (every second): ######### host:127.0.0.1,port:54087 accept_cb: cli:0x9f15c00, input_header:0x9e6bfc0 host=127.0.0.1,port=54087:python_handler:HEADER: ø³c· bð ** host=127.0.0.1,port=54087 write_cli:uri=(null),r=165,len=165,c=1 host=127.0.0.1,port=54087 close_connection:cli:0x9f15c00, input_header:0x9e6bfc0*** #########

these request maybe not handled properly by fapws3 v0.10-11 the problem solved after i upgrade to fapws3 v0.10-12, hooraaay :D

in fapws3 version v0.10-11, user should define httpchk option uri in haproxy side, as i mentioned before. otherwise haproxy will requesting HEAD with null uri which (maybe) unhandled by fapws3.

Thank you for your support.

william-os4y commented 12 years ago

Good to know :-)

On my machine, I've never had any issues with the versions you are talking about. But, indeed it sounds that my "realloc" changes were not so good :-(.

Thanks to others tests, I'll propose a new release. Better for people using Pypi.

Thanks

On Thu, Apr 12, 2012 at 6:16 AM, wigunawan reply@reply.github.com wrote:

William, after i set debug mode in fapws3, i see that haproxy send following request periodically (every second): ######### host:127.0.0.1,port:54087 accept_cb: cli:0x9f15c00, input_header:0x9e6bfc0 host=127.0.0.1,port=54087:python_handler:HEADER: ø³c· bð ** host=127.0.0.1,port=54087 write_cli:uri=(null),r=165,len=165,c=1 host=127.0.0.1,port=54087 close_connection:cli:0x9f15c00, input_header:0x9e6bfc0*** #########

these request maybe not handled properly by fapws3 v0.10-11 the problem solved after i upgrade to fapws3 v0.10-12, hooraaay :D

in fapws3 version v0.10-11, user should define httpchk option uri in haproxy side, as i mentioned before. otherwise haproxy will requesting HEAD with null uri which (maybe) unhandled by fapws3.

Thank you for your support.

Reply to this email directly or view it on GitHub: https://github.com/william-os4y/fapws3/issues/36#issuecomment-5084286