[tempo-distributed] gateway does not support all protocols

[BitProcessor]

It seems to me that the gateway as introduced by this PR: https://github.com/grafana/helm-charts/pull/679 does not support all protocols:

https://github.com/grafana/helm-charts/pull/679/files#diff-5414418aa29e8ed64794c82c7095aa92f5086272f5bd83c823d2c97c7d108d31R699-R733

For example, otlp-grpc on port 4317 is not supported It's a bit silly to allow configuration of it in the traces block:

    traces:
      otlp:
        grpc: true

But next, don't configure the gateway (and ingress) to support it.

Confusing + not a great default experience.

[ahaw023]

@BitProcessor -- it is a little daft.

I am using an OpenTelemetry collector and this is the config i set up to make it work

    exporters:
       otlphttp/grafana-hosted:
        endpoint: http://tempo-distributed-gateway.grafana-tempo.svc.cluster.local:80/otlp

[ggogel]

@ahaw023 This is not a fix for the problem. You need the gateway + ingress to receive data from an app outside the cluster running tempo, e.g., in a multi-cluster environment. Your proposed solution only works within the same cluster. You don't even need a gateway here.

I think that the only workaround at the moment is to implement an ingress object that points to the tempo-distributor service.

[aporwal3]

@ggogel @BitProcessor @ahaw023 @bcotton Can you please help on the ingress yaml that you setup to make it work as workaround.

We have setup following ingress and it did not worked.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: tempo-distributor-ingress
  namespace: monitoring
spec:
  rules:
  - host: tempo-ingress.exm.com
    http:
      paths:
      - backend:  
          service:
            name: lgtm-tempo-distributor
            port:
              number: 4317
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - tempo-ingress.exm.com
    secretName: tls

On the application properties set up the following to send the traces

-Dquarkus.otel.exporter.otlp.traces.endpoint=https://tempo-ingress.exm.com

We also tried using AWS NLB and ALB and they are also not working with the same. Please advise.