Issues with gRPC Communication between AnyCable Go and AnyCable Rails

[pritambios]

I am new to Anycable and grpc. AnyCable is up and running smoothly on my local machine. Now, I'm looking to deploy it to AWS environment.

Background I have an AnyCable Go service running on a dedicated EC2 machine. The AnyCable Rails gRPC server is on a different machine, operating on port 50051, with a health check server running at port 54321. *I have introduced a Network Load Balancer (NLB) to facilitate communication between the two.

Current Status From AnyCable Go host machine I am successfully able to checks the health of the AnyCable Rails gRPC server using port 54321. However, it's encountering difficulties in making gRPC calls to port 50051. Error: anycable-go[121455]: E 2023-11-22T09:30:44.247Z sid=HHfjSvRIk79FjN82LOB4u WebSocket session failed: grpc connection is not ready

Actions Taken: Security groups have been configured to allow traffic on both port 50051 and 54321 on the AnyCable Rails machine for anycable-go machine. NLB configured to check the health of the AnyCable Rails server. And in NLB, grpc instances are healthy.

Despite these configurations, I am unable to establish gRPC communication on port 50051.

My Anycable go server systemd config

[Unit]
Description=AnyCable Go WebSocket Server
After=network.target

[Service]
Type=simple
ExecStart=/home/ec2-user/go/bin/anycable-go --headers=cookie,authorization
ExecStop=/bin/kill -TERM $MAINPID
LimitNOFILE=16384
Restart=on-failure

Environment=ANYCABLE_HOST=localhost
Environment=ANYCABLE_PORT=8080
Environment="ANYCABLE_REDIS_URL=redis://************:6379?timeout=2"
Environment="ANYCABLE_RPC_HOST=awseb-network-load-balancer-dns.amazonaws.com:50051"
Environment=ANYCABLE_PATH=/cable
Environment=ANYCABLE_REDIS_CHANNEL=__anycable__

[Install]
WantedBy=multi-user.target

Ruby version: 3.0.6 Rails version: 6.1.5 anycable gem version: 1.4.3 anycable-rails gem version: 1.4.2 grpc gem version: 1.59.2

[palkan]

Hey @pritambios! Have you figured out what was the issue?

[pritambios]

I successfully resolved the issue by carefully following the AnyCable load balancer documentation(Link). The problem stemmed from my Nginx configuration.