`403 Forbidden` from API endpoints

[nforro]

For a while dashboard has been having issues with rejected API requests because of missing Access-Control-Allow-Origin header.

I have to refresh multiple times (it varies, but usually more than 10 times) until an API request succeeds and dashboard becomes usable - once a requests succeeds, the problem goes away until I close the browser tab.

It happens across different OSes with different browser profiles, I use chromium-based browser but I'm quite sure it happened in Firefox as well, though I'm not able to reproduce it with Firefox now.

[Venefilyn]

@nforro this has to do with the API missing the header when returning 403. It's nothing to do with the dashboard. Why it's returning 403 I couldn't tell you. Maybe it's a specific pod that is unavailable and returning 403 Forbidden?

[nforro]

Makes sense, so there are two issues, one similar to https://github.com/packit/packit-service/issues/1883 and the other needs further investigation.

[nforro]

Here is a request sent from the browser (failed ones and successful ones are exactly the same):

fetch("https://stg.packit.dev/api/copr-builds?page=1", {
  "headers": {
    "accept": "*/*",
    "accept-language": "cs-CZ,cs;q=0.9,en;q=0.8",
    "priority": "u=1, i",
    "sec-ch-ua": "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-site"
  },
  "referrer": "https://dashboard.stg.packit.dev/",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": null,
  "method": "GET",
  "mode": "cors",
  "credentials": "omit"
});

When the request is successful, it's visible in service logs:

172.20.122.9 - - [02/Oct/2024:19:33:50 +0000] "GET /api/copr-builds?page=1 HTTP/2.0" 206 8640

When the request ends with 403 Forbidden, there is nothing in service logs, however there is this in dashboard logs:

[Wed Oct 02 19:31:33.898739 2024] [authz_core:error] [pid 51:tid 62] [remote 172.20.122.9:33218] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8000:1007890000/htdocs/api, referer: https://dashboard.stg.packit.dev/
172.20.122.9 - - [02/Oct/2024:19:31:33 +0000] "GET /api/copr-builds?page=1 HTTP/2.0" 403 199

[Venefilyn]

Usually HEADER requests doesn't show in logs IIRC. Can you enable verbose logging for the staging instance to investigate more?

This sounds like a configuration issue with the mod_wsgi settings

client denied by server configuration: /tmp/mod_wsgi-localhost:8000:1007890000/htdocs/api

Possibly related? https://stackoverflow.com/questions/16362462/flask-mod-wsgi-client-denied-by-server-configuration

[mfocko]

But there should be no requests to the dashboard itself, everything uses Service API

[nforro]

Usually HEADER requests doesn't show in logs IIRC. Can you enable verbose logging for the staging instance to investigate more?

I don't think the browser sends any HEAD requests, but more verbose logging on stage can't hurt, I guess --log-level info should do the trick.

[Venefilyn]

Ah huh.. it shouldn't make any requests at all to the dashboard, particularly because I've setup a hard-code for all React API requests to use the import.meta.env.VITE_API_URL, which is set to https://prod.packit.dev/api for prod and it defaults to staging if not set

So it shouldn't direct it to the dashboard. Maybe there is an routing ingress issue in OCP that is not explicitly set?

[nforro]

it shouldn't make any requests at all to the dashboard, particularly because I've setup a hard-code for all React API requests to use the import.meta.env.VITE_API_URL, which is set to https://prod.packit.dev/api for prod and it defaults to staging if not set

Yes, the requests are made to stg.packit.dev, even the :authority: pseudo-header is set to stg.packit.dev. Of course it happens on prod too.

[mfocko]

I don't see anything wrong, so reaching out to the cluster admins.

Reference: https://redhat-internal.slack.com/archives/C04LZLPE7QU/p1730903564686669

[mfocko]

Ticket: RITM1933277