Cors errors with Grafana Agent

nlamirault commented 7 months ago

Hi, i would like to test Faro with Grafana Agent with this application. I use a K3s cluster with nginx-ingress-controller deployed, and a Grafana Agent with Faro configuration:

...

faro.receiver "LABEL" {
  server {
    listen_address = "0.0.0.0"
    cors_allowed_origins = [
      "*",
    ]
  }

  output {
    logs   = [
      loki.process.faro_logfmt.receiver,
    ]
    traces = [
      otelcol.exporter.otlp.local.input,
    ]
  }
}

...

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-methods: GET, PUT, POST, DELETE, PATCH,
      OPTIONS
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/cors-max-age: "1728000"
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: grafana-agent-frontend-faro
    app.kubernetes.io/instance: grafana-agent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana-agent-frontend
    app.kubernetes.io/version: v0.37.4
    argocd.argoproj.io/instance: grafana-agent
    helm.sh/chart: grafana-agent-frontend-0.27.2
  name: agent-frontend-faro-external
  namespace: opentelemetry
spec:
  ingressClassName: nginx
  rules:
  - host: agent-frontend-faro.home.portefaix.xyz
    http:
      paths:
      - backend:
          service:
            name: grafana-agent-grafana-agent-frontend-faro
            port:
              number: 12347
        path: /
        pathType: Prefix

Pizza configuration:

apiVersion: v1
data:
  QUICKPIZZA_CONF_FARO_URL: http://agent-frontend-faro.home.portefaix.xyz
  QUICKPIZZA_LOG_LEVEL: debug
  QUICKPIZZA_OTLP_ENDPOINT: http://grafana-agent-grafana-agent-traces.opentelemetry.svc.cluster.local:4318
kind: ConfigMap
metadata:
  labels:
    app.k8s.io/name: quickpizza
    app.kubernetes.io/component: config
    app.kubernetes.io/instance: tracing-env
    argocd.argoproj.io/instance: pizza-dev
  name: tracing-env
  namespace: demo-dev

Pizza application is deployed and works fine. But in the browser console, i've got these logs:

pizza-dev.home.portefaix.xyz/:1 Access to fetch at 'http://agent-frontend-faro.home.portefaix.xyz/' from origin 'http://pizza-dev.home.portefaix.xyz' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
start.27e40fd3.js:1 

POST http://agent-frontend-faro.home.portefaix.xyz/ net::ERR_FAILED 405 (Method Not Allowed)
window.fetch @ start.27e40fd3.js:1
(anonymous) @ app.000e2920.js:55
r.<computed> @ app.000e2920.js:48
(anonymous) @ app.000e2920.js:32
invoke @ app.000e2920.js:42
run @ app.000e2920.js:42
e.with @ app.000e2920.js:32
e.with @ app.000e2920.js:32
(anonymous) @ app.000e2920.js:32
p @ app.000e2920.js:55
(anonymous) @ app.000e2920.js:32
(anonymous) @ app.000e2920.js:31
o @ app.000e2920.js:1
(anonymous) @ app.000e2920.js:1
p @ app.000e2920.js:55
add @ app.000e2920.js:1
(anonymous) @ app.000e2920.js:31
(anonymous) @ app.000e2920.js:31
p @ app.000e2920.js:55
Nu @ app.000e2920.js:31
send @ app.000e2920.js:31
f @ app.000e2920.js:4
flush @ app.000e2920.js:1
(anonymous) @ app.000e2920.js:1
_.<computed> @ app.000e2920.js:91
invokeTask @ app.000e2920.js:42
runTask @ app.000e2920.js:42
invokeTask @ app.000e2920.js:42
x.useG.invoke @ app.000e2920.js:42
l.args.<computed> @ app.000e2920.js:91
setInterval (async)
s @ app.000e2920.js:91
scheduleTask @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleMacroTask @ app.000e2920.js:42
li @ app.000e2920.js:48
(anonymous) @ app.000e2920.js:91
r.<computed> @ app.000e2920.js:48
start @ app.000e2920.js:1
cs @ app.000e2920.js:1
ds @ app.000e2920.js:5
xs @ app.000e2920.js:29
Mu @ app.000e2920.js:32
(anonymous) @ app.000e2920.js:109
invoke @ app.000e2920.js:42
run @ app.000e2920.js:42
(anonymous) @ app.000e2920.js:55
invokeTask @ app.000e2920.js:42
runTask @ app.000e2920.js:42
v @ app.000e2920.js:42
Promise.then (async)
I @ app.000e2920.js:42
N @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleMicroTask @ app.000e2920.js:42
Te @ app.000e2920.js:55
le @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
Promise.then (async)
(anonymous) @ app.000e2920.js:55
p @ app.000e2920.js:55
O.then @ app.000e2920.js:55
le @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:55
invokeTask @ app.000e2920.js:42
runTask @ app.000e2920.js:42
v @ app.000e2920.js:42
Promise.then (async)
I @ app.000e2920.js:42
N @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleMicroTask @ app.000e2920.js:42
Te @ app.000e2920.js:55
le @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
Promise.then (async)
(anonymous) @ app.000e2920.js:55
p @ app.000e2920.js:55
O.then @ app.000e2920.js:55
nE @ app.000e2920.js:109
(anonymous) @ app.000e2920.js:109
app.000e2920.js:5 Faro
 @grafana/faro-web-sdk:transport-fetch
 Failed sending payload to the receiver
 {meta: {…}, events: Array(2), measurements: Array(2)} TypeError: Failed to fetch
    at window.fetch (start.27e40fd3.js:1:1545)
    at app.000e2920.js:55:3164
    at r.<computed> (app.000e2920.js:48:3547)
    at app.000e2920.js:32:123537
    at u.invoke (app.000e2920.js:42:6431)
    at s.run (app.000e2920.js:42:1820)
    at e.with (app.000e2920.js:32:136111)
    at e.with (app.000e2920.js:32:12794)
    at app.000e2920.js:32:123455
    at new p (app.000e2920.js:55:2172)

How can i fix that CORS error ? Thanks.

roobre commented 7 months ago

Hi @nlamirault,

Your setup looks alright to me at first glance, but I would suggest some additional troubleshooting steps to see what's going on. The No 'Access-Control-Allow-Origin' header is present seems to hint that the ingress config is not materializing as expected. I would suggest to try the following:

Check that you don't have any other ingress set up for that host:

kubectl get ingresses -A -o wide | grep -ie agent-frontend-faro

Curl the faro endpoint from the same machine running the browser and see what headers are returned:

curl -v http://agent-frontend-faro.home.portefaix.xyz
Maybe try a POST as well curl -v -d '{}' http://agent-frontend-faro.home.portefaix.xyz

Check the logs for your ingress controller and verify that you can see the request you make with both curl and the browser is correctly received. nginx also logs the service and endpoint backing the request, you can check it matches grafana-agent-grafana-agent-frontend-faro.

In the Network tab of your browser's development tools, inspect the request Faro is doing. You won't be able to see the response (it is blocked by cors), but the request might be worth checking, in case it includes strange headers.

Right click that request, "Copy as curl", and try to issue it in the terminal. There you will be able to see the response with its headers.

If this does not lead you to the problem, feel free to paste the output of those commands here, and we'll do our best to help you diagnose the situation :)

nlamirault commented 1 week ago

Hi @roobre i migrate to Alloy. I've got 2 ingress:

one for Alloy UI (which works fine)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/cors-allow-headers: DNT,X-CustomHeader,X-LANG,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-Api-Key,X-Device-Id,Access-Control-Allow-Origin
nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE
nginx.ingress.kubernetes.io/cors-allow-origin: '*'
nginx.ingress.kubernetes.io/enable-cors: "true"
labels:
app: alloy-frontend
app.kubernetes.io/instance: alloy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: alloy-frontend
app.kubernetes.io/part-of: alloy
app.kubernetes.io/version: v1.1.0
argocd.argoproj.io/instance: alloy
helm.sh/chart: alloy-frontend-0.3.0
portefaix.xyz/version: v0.54.0
name: alloy-frontend-external
namespace: opentelemetry
spec:
ingressClassName: nginx
rules:
- host: alloy-frontend.home.portefaix.xyz
http:
  paths:
  - backend:
      service:
        name: alloy-alloy-frontend
        port:
          number: 12345
    path: /
    pathType: Prefix

one for Alloy with Faro (on 12347)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/cors-allow-methods: GET, PUT, POST, DELETE, PATCH,
  OPTIONS
nginx.ingress.kubernetes.io/cors-allow-origin: '*'
nginx.ingress.kubernetes.io/cors-max-age: "1728000"
nginx.ingress.kubernetes.io/enable-cors: "true"
labels:
app: alloy-frontend
app.kubernetes.io/instance: alloy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: alloy-frontend
app.kubernetes.io/part-of: alloy
app.kubernetes.io/version: v1.1.0
argocd.argoproj.io/instance: alloy
helm.sh/chart: alloy-frontend-0.3.0
portefaix.xyz/version: v0.54.0
name: alloy-frontend-faro-external
namespace: opentelemetry
spec:
ingressClassName: nginx
rules:
- host: alloy-frontend-faro.home.portefaix.xyz
http:
  paths:
  - backend:
      service:
        name: alloy-alloy-frontend
        port:
          number: 12347
    path: /
    pathType: Prefix

and I've got an error using curl:

$ curl -v -d '{}' http://alloy-frontend-faro.home.portefaix.xyz
* Host alloy-frontend-faro.home.portefaix.xyz:80 was resolved.
* IPv6: (none)
* IPv4: 100.126.241.86
*   Trying 100.126.241.86:80...
* Connected to alloy-frontend-faro.home.portefaix.xyz (100.126.241.86) port 80
> POST / HTTP/1.1
> Host: alloy-frontend-faro.home.portefaix.xyz
> User-Agent: curl/8.6.0
> Accept: */*
> Content-Length: 2
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 405 Method Not Allowed
< Date: Fri, 21 Jun 2024 14:52:14 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 18
< Connection: keep-alive
< Allow: GET, OPTIONS
<
* Connection #0 to host alloy-frontend-faro.home.portefaix.xyz left intact
Method Not Allowed%

roobre commented 5 days ago

Hey,

To me something seems to be off with your ingress-nginx annotations, or config. The curl command you posted should have returned CORS headers, something like:

< access-control-allow-origin: *
< access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
< access-control-allow-headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization

I'd suggest taking a look at the ingress-nginx manuals, logs, and/or generated config to try and figure out what's happening in there.

grafana / quickpizza

Cors errors with Grafana Agent #47