nginx 转发 https 协议的higress 域名出现104 Connection reset by peer报错

[whitebear009]

有一个 nginx 配置了 proxy_pass 转发到了一个 ingress 域名，我把这个 ingress 从 nginx ingress 切换到了 higress 之后就不能工作了。 nginx 的完整配置如下：

/ # cat /etc/nginx/config/nginx.dev
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   180;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    client_header_buffer_size 512k;
    large_client_header_buffers 4 512k;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location ~* /(api/|qhinternalssostationpage.html) {
            proxy_pass https://customplatform-dev.qunhequnhe.com;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /docs {
            index index.html;
        }

        location / {
            index index.html;
            try_files $uri /index.html;
        }
    }
}

customplatform-dev.qunhequnhe.com 是一个 higress gateway 的域名，这里如果配置的是 https，nginx 的 access.log 会报错：

2024/07/10 09:32:09 [error] 52#52: *1738 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 172.28.181.31, server: _, request: "POST /ctplatform/api/log/track HTTP/1.1", upstream: "https://10.101.2.91:443/ctplatform/api/log/track", host: "param-platform-ui-sit.qunhequnhe.com", referrer: "https://param-platform-ui-prod.qunhequnhe.com/"

如果配置的是 http 则可以正常访问。而且在 nginx 里 curl https 的这个域名也可以正常返回 200。

看起来跟这个 issue （ https://github.com/alibaba/higress/issues/651 ）比较像，但貌似没有提供解决方法。 希望大佬们帮忙瞧瞧。

[whitebear009]

查了一下发现在这个 nginx 上加一条 proxy_ssl_server_name on; 就可以了。

但还是不太理解为什么原来当 https://customplatform-dev.qunhequnhe.com 是 nginx ingress 域名的时候是可以工作的

[johnlanni]

这个就是这样proxy pass的时候nginx默认不传sni。 nginx ingress会默认装一本证书用于没有匹配sni的场景，higress不会，需要给一个没有指定host的ingress设置一个tls secret才可以。

[whitebear009]

需要给一个没有指定host的ingress设置一个tls secret才可以

指的是这个注解是吗 higress.io/proxy-ssl-secret

[johnlanni]

不是，ingress spec里的secret字段

[whitebear009]

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: empty-sni
  namespace: ingress-service
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: empty-sni-svc
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - secretName: https-secret-qunhequnhe

---
apiVersion: v1
kind: Service
metadata:
  name: empty-sni-svc
  namespace: ingress-service
spec:
  externalName: higress-gateway.higress-system.svc.cluster.local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: ExternalName

加了这一段 yaml 后确实能工作了，大佬说的是这个意思吗