Gateway HA - Client unable to make connection to Server agents

[ramaraosrikakulapu]

This is part of Gateway HA and related to the issue.

Scenario:

• Gateway load balancer is running in VM with IP x.x.1.124
• Gateway instances are running in 2 VM's with hst URL - ws://x.x.1.124/agent
• Both gateway instances are registered with EC Service with diff refId's
• Server agent running fine and made super connection with both gateways (-rht localhost and -rpt 22)
• Gateway health page looks good and able to see superconns for both gateway instances
• Client started running fine
• Client request is picked by one gateway and reaches to the server, but could not able to establish the connection

[ramaraosrikakulapu]

Here are the logs for ref -

Client initiates the request -

[EC Client x0] 2020/07/17 10:42:02 [Rev: v1.1beta.fukuoka.2737]
 [EC Client x0] 2020/07/17 10:42:02 [Agent websocket compression is being negotiated at level 0]
 [EC Client x0] 2020/07/17 10:42:02 [refresh duration is specified. Token auto-refresh is set.]
 [EC Client x0] 2020/07/17 10:42:02 [Fetching new token..]
 [EC Client x0] 2020/07/17 10:42:06 [Token refreshed. The token will be expired in 719 minutes. Approx. 20 minutes to the next auto-refresh.]
 [EC Client x0] 2020/07/17 10:42:06 [health api will be available on the local port#:7991]
 [EC Client x0] 2020/07/17 10:42:06 [github.com/wzlib/wzcore.(*Client).Hire.func5.1 listen tcp :7991: bind: address already in use]
 [EC Client x0] 2020/07/17 10:42:06 [[VLAN] client is listening on port# :6193]

 [EC Client x0] 2020/07/17 10:42:20 [http proxy was not used for the request: http://x.x.1.124/agent]
 [EC Client x0] 2020/07/17 10:42:28 [client#r36L4J connection is pending for a confirmation from the gateway]
 [EC Client x0] 2020/07/17 10:42:28 [connection request has been received by the gateway with the bindId#FJYjXxqUJ6]
 [EC Client x0] 2020/07/17 10:42:40 [agent exiting TCP_REQUEST.]
 [EC Client x0] 2020/07/17 10:42:40 [bind#: FJYjXxqUJ6 has been removed from the pool.]

Gateway1 picked the request -

[EC Gateway x0] 2020/07/17 14:42:20 [the request port might not be specified. continuing validation..]
 [EC Gateway x0] 2020/07/17 14:42:20 [ip passed filtering with x-forward-for.]
 [EC Gateway x0] 2020/07/17 14:42:24 [a superconn is identified for server agent#0jhqPu]
 [EC Gateway x0] 2020/07/17 14:42:28 [client acknowledged the connection. requesting server callback with bindId: FJYjXxqUJ6, clientId: r36L4J, gatewayKey: 08:00:27:67:03:cf:x0]
 [EC Gateway x0] 2020/07/17 14:42:40 [client (bind# FJYjXxqUJ6) pending longer than expected, is now being removed from the pool.]
 [EC Gateway x0] 2020/07/17 14:42:40 [exception occurred in client reading. err: read tcp x.x.1.121:17990->x.x.1.124:50860: use of closed network connection  (session#   bind# FJYjXxqUJ6 )]
 [EC Gateway x0] 2020/07/17 14:42:40 [client with bindId:FJYjXxqUJ6has been removed from the pools.]
 [EC Gateway x0] 2020/07/17 14:42:40 [Both sessionId:  and bindId: FJYjXxqUJ6 have been removed from the pools.]
 [EC Gateway x0] 2020/07/17 14:42:40 [client with bindId:FJYjXxqUJ6has been removed from the pools.]
 [EC Gateway x0] 2020/07/17 14:42:47 [reporting usage..]
 [EC Gateway x0] 2020/07/17 14:42:48 [usage reported.]
 [EC Gateway x0] 2020/07/17 14:42:48 [websocket: close 1006 (abnormal closure): unexpected EOF]
 [EC Gateway x0] 2020/07/17 14:42:48 [support connection for Server Agent Id#0jhqPu has been deleted.]

Gateway2 logs just for ref -

[EC Gateway x0] 2020/07/17 14:42:28 [ip passed filtering with x-forward-for.]
 [EC Gateway x0] 2020/07/17 14:42:31 [received server# 0jhqPu request]
 [EC Gateway x0] 2020/07/17 14:42:48 [websocket: close 1006 (abnormal closure): unexpected EOF]
 [EC Gateway x0] 2020/07/17 14:42:48 [support connection for Server Agent Id#0jhqPu has been deleted.]

Server logs -

[EC Server x0] 2020/07/17 10:42:08 [initialising super connection for gateway inst#x0]
 [EC Server x0] 2020/07/17 10:42:08 [http proxy was not used for the request: http://x.x.1.124/agent]
 [EC Server x0] 2020/07/17 10:42:08 [http proxy was not used for the request: http://x.x.1.124/agent]
 [EC Server x0] 2020/07/17 10:42:11 [super connection established for gateway inst#x0]
 [EC Server x0] 2020/07/17 10:42:12 [super connection established for gateway inst#x0]
[EC Server x0] 2020/07/17 10:42:28 [client#r36L4J requests connection with the bind id#FJYjXxqUJ6.]
 [EC Server x0] 2020/07/17 10:42:28 [http proxy was not used for the request: http://x.x.1.124/agent]
 [EC Server x0] 2020/07/17 10:42:31 [server has been successfully created for clientId#r36L4J with bindId#FJYjXxqUJ6]
 [EC Server x0] 2020/07/17 10:42:31 [server standby in the gateway with session#]

[ramaraosrikakulapu]

@ayasda2003 .. right now I am not not using any headers in lber configuration for redirecting the traffic to a specific gateway from load balancer.

While forwarding the client request to server agent, load balancer might not remember which gateway holds the client request and while server responding back, server agent should connect to same gateway and looks its not happening. Using a header can solve the problem?

Thanks @palokam for sharing your thoughts.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ramaraosrikakulapu]

Closing the issue as Gateway HA implemented and in use by customers