search

fabiolb / fabio

Consul Load-Balancing made simple
https://fabiolb.net
MIT License
7.26k stars 616 forks source link

Fabio on gRPC mode closes open streams on route update even if the same route is added. #809

Open cmvoicu opened 3 years ago

cmvoicu commented 3 years ago

Hi,

It seems that fabio closes open gRPC stream when a route update happens. Also unfortunatelly not sure why i can't set the level to debug:

2020/12/22 00:02:01 [INFO] Cannot set log level to DEBUG

2020/12/22 00:02:01 [INFO] Version 1.5.11 starting 2020/12/22 00:02:01 [INFO] Go runtime is go1.11.5 2020/12/22 00:02:01 [INFO] Metrics disabled 2020/12/22 00:02:01 [INFO] Setting GOGC=800 2020/12/22 00:02:01 [INFO] Setting GOMAXPROCS=32

2020/12/22 02:40:51 [WARN] consul: Error getting catalog service rtp-prd-position-service. Unexpected response code: 500 (rpc error making call: rpc error making call: node is not the leader) 2020/12/22 02:40:51 [INFO] Config updates -route add rtp-prd-position-service /business.RTPoseService grpcs://climatologue.fr.world:7772 opts "tlsskipverify=true" -route add rtp-prd-position-service /business.RTPoseService grpcs://cardiologue.fr.world:7771 opts "tlsskipverify=true" 2020/12/22 02:40:54 [INFO] Config updates +route add rtp-prd-position-service /business.RTPoseService grpcs://climatologue.fr.world:7772 opts "tlsskipverify=true" +route add rtp-prd-position-service /business.RTPoseService grpcs://cardiologue.fr.world:7771 opts "tlsskipverify=true"

Any idea why this happens...it can happen like 2-3 times per day. Maybe because of the consul error retrieving catalog service? Here is the file config used: proxy.cs = cs=machine;type=file;.... proxy.addr = :8888;proto=grpcs;cs=machine ui.addr = xxx.xx.xx.xx ui.access =rw ui.title = rtpose-fr-prd log.level = DEBUG log.access.target = stdout log.access.level = DEBUG registry.consul.register.enabled = true registry.consul.register.name = rtp-prd-fabio registry.consul.register.addr = :9999 registry.consul.token = registry.consul.pollinterval =15s registry.consul.tls.insecureskipverify =true proxy.maxconn = 20000

Thank you!

qqwx1986 commented 3 years ago

yes , it will close open grpc stream connect when update route, because when a route update it will create a new Table but function hasTarget compared by table pointer. I solved in #807

cmvoicu commented 3 years ago

hi, thanks for your answer...i see the fix is in PR, not yet merged...is there a workarround for this with configs, as it seems the issue for not getting the catalog service is transiet(it works later on)....so should i decrease the config updates/pollInterval from consul or increase the 5s grpcConnectionPool cleanup, some number of retries or something?

qqwx1986 commented 3 years ago

hi, thanks for your answer...i see the fix is in PR, not yet merged...is there a workarround for this with configs, as it seems the issue for not getting the catalog service is transiet(it works later on)....so should i decrease the config updates/pollInterval from consul or increase the 5s grpcConnectionPool cleanup, some number of retries or something?

i'm sorry i think there is no config can solve this issue