search

Xyphyn / photon

A replacement for lemmy-ui with more features, a better design, and more customizability.
https://phtn.app
GNU Affero General Public License v3.0
344 stars 38 forks source link

Photon 1.27.x does not work witth HAProxy - fails http healthchecks #267

Open nsakkos opened 6 months ago

nsakkos commented 6 months ago

Describe the bug Since v1.27.0 Photon doesn't work with HAProxy, because it fails the basic http healthcheck:

backend photon
  option httpchk
  timeout check 15s
  server photon photon:3000 check

I have tried the latest version of Photon and it still fails.

HAProxy shows this in the logs:

2024-03-02T20:09:23.877695000-05:00 [WARNING]  (3) : Server photon/photon is DOWN, reason: Layer7 invalid response, check duration: 10670ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2024-03-02T20:09:23.878514000-05:00 [ALERT]    (3) : backend 'photon' has no server available!

tcpdump reveals that after HAProxy sends a GET request, Photon sends an ACK, but no other payload. Photon terminates the connection after about 10 seconds:


02:19:13.513725 IP haproxy-dev > photon: Flags [S], seq 3798280417, win 64860, options [mss 1410,sackOK,TS val 2675610512 ecr 0,nop,wscale 7], length 0
02:19:13.516771 IP photon > haproxy-dev: Flags [S.], seq 907003150, ack 3798280418, win 65160, options [mss 1460,sackOK,TS val 464693127 ecr 2675610512,nop,wscale 7], length 0
02:19:13.516968 IP haproxy-dev > photon: Flags [P.], seq 1:19, ack 1, win 507, options [nop,nop,TS val 2675610515 ecr 464693127], length 18
02:19:13.519792 IP photon > haproxy-dev: Flags [.], ack 19, win 509, options [nop,nop,TS val 464693130 ecr 2675610515], length 0
02:19:23.512057 IP photon > haproxy-dev: Flags [F.], seq 1, ack 19, win 509, options [nop,nop,TS val 464703122 ecr 2675610515], length 0
02:19:23.512426 IP haproxy-dev > photon: Flags [F.], seq 19, ack 2, win 507, options [nop,nop,TS val 2675620510 ecr 464703122], length 0
02:19:23.515316 IP photon > haproxy-dev: Flags [.], ack 20, win 509, options [nop,nop,TS val 464703125 ecr 2675620510], length 0
02:19:25.515197 IP haproxy-dev > photon: Flags [S], seq 3087459844, win 64860, options [mss 1410,sackOK,TS val 2675622513 ecr 0,nop,wscale 7], length 0
02:19:25.518304 IP photon > haproxy-dev: Flags [S.], seq 366076850, ack 3087459845, win 65160, options [mss 1460,sackOK,TS val 464705128 ecr 2675622513,nop,wscale 7], length 0
02:19:25.518510 IP haproxy-dev > photon: Flags [P.], seq 1:19, ack 1, win 507, options [nop,nop,TS val 2675622516 ecr 464705128], length 18
02:19:25.521475 IP photon > haproxy-dev: Flags [.], ack 19, win 509, options [nop,nop,TS val 464705131 ecr 2675622516], length 0
02:19:35.512039 IP photon > haproxy-dev: Flags [F.], seq 1, ack 19, win 509, options [nop,nop,TS val 464715122 ecr 2675622516], length 0
02:19:35.512447 IP haproxy-dev > photon: Flags [F.], seq 19, ack 2, win 507, options [nop,nop,TS val 2675632510 ecr 464715122], length 0
02:19:35.515473 IP photon > haproxy-dev: Flags [.], ack 20, win 509, options [nop,nop,TS val 464715125 ecr 2675632510], length 0

Checking connectivity to Photon with curl does not cause the same issue, Photon responds with a payload.

To Reproduce (if applicable) Steps to reproduce the behavior:

  1. Place the attached docker-compose.yaml and haproxy.cfg in the same folder (remove .txt extensions)
  2. docker compose up -d
  3. docker logs -tf haproxy

Expected behavior The expected behavior would be for photon to pass the HAProxy http healthchecks, which were working up to Photon v1.26.0, so that HAProxy can route traffic to healthy photon instances.

Screenshots If applicable, add screenshots to help explain your problem.

Versions

*HAProxy 2.7.11 gives a slightly different message "Server photon/photon is DOWN, reason: Layer7 invalid response, info: "TCPCHK got an empty response at step 2", check duration: 10736ms." haproxy.cfg.txt docker-compose.yaml.txt

CTalvio commented 6 months ago

253 related?

Something in 1.27.0 changed, and it broke using photon with nginx as well, at least for some deployments.