Server behind Istio Gateway

Pre-requisites

[X] I have double-checked my configuration
[X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[X] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Downloaded, installed Argo Workflows 3.5.10, with secure=false and BASE_HREF=/ behind Istio, gives errors of:

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure

Or, it just loads a blank page

Version(s)

3.5.10

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

No workflows yet being that I wasn't even able to get to that part

What I can tell you is that, I have set:

--secure false with BASE_HREF=/.

On Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: dev-gateway
spec:
  selector:
    istio: dev-ingressgateway
  servers:
  - hosts:
    - workflows.mycompany.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: istio-ingress-server-tls
      mode: SIMPLE

VirtualService:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: argo-vs
  namespace: argo
spec:
  gateways:
  - default/dev-gateway
  hosts:
  - workflows.mycompany.com
  http:
  - match:
    - uri:
        prefix: /
    rewrite:
      uri: /
    route:
    - destination:
        host: my-argo-workflows-server
        port:
          number: 2746

Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: server
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: my-argo-workflows
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argo-workflows-server
    app.kubernetes.io/part-of: argo-workflows
    app.kubernetes.io/version: v3.5.10
    helm.sh/chart: argo-workflows-0.41.14
  name: my-argo-workflows-server
  namespace: argo
spec:
  clusterIP: 172.16.141.165
  clusterIPs:
  - 172.16.141.165
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 2746
    protocol: TCP
    targetPort: 2746
  type: ClusterIP
status:
  loadBalancer: {}

Logs from the workflow controller

I haven't ran any workflows yet

Logs from in your workflow's wait container

I haven't ran any workflows yet

I also took a look and did some searching and saw something similar here: https://github.com/argoproj/argo-workflows/issues/3080#issuecomment-1278072332 from @arnoldrw and @agilgur5 , I tried that but to no avail.

argoproj / argo-workflows

Server behind Istio Gateway #13449