`403 Forbidden` errors in `cves` command

velll commented 2 months ago

Hi everyone! 👋

We are using the docker/scout-action in our CD pipeline, and we started getting 403 errors from the cves command today. The error looks like this:

Error: API operation failed: Message: 403 Forbidden; body: "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>403 Forbidden</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Forbidden</h1>\n<h2>Your client does not have permission to get URL <code>/v1/graphql</code> from this server.</h2>\n<h2></h2>\n</body></html>\n", Locations: [], Extensions: map[code:request_error], Path: []

This does not happen on every image. I am getting cves for some of our images (logs, comments, the works), and errors for others. Couldn't find any system. This makes me think that it's not an issue with our secrets (PAT).

❓ Could you help me understand what could lead to this error? Which service does the action tries to reach in this case? (Our registry, something internal in scout, github? What should I double check in our access credentials?)

What I tried?

I tried running docker scout cves with the same configuration on the same images locally on v1.9.3 — the command runs, and I am receiving correct output in all cases. This makes me think the error is related to the actions specifically. 🤔

I also tried setting the version of the action to v1.9.1, this did not change the error.

Runners

We run our actions on self-hosted x86 ubuntu

Output

Here is the full output of the action including the configuration

Run docker/scout-action@v1
  with:
    command: cves
    dockerhub-user: ***
    dockerhub-password: ***
    image: ***redacted***
    only-severities: critical,high
    exit-code: false
    only-fixed: true
    summary: true
    format: json
    github-token: ***
    write-comment: true
/usr/bin/docker exec  ***there_was_a_long_hash_here*** sh -c "cat /etc/*release | grep ^ID"
cves
      ...Pulling
      ✓ Pulled
      ...Storing image for indexing
      ✓ Image stored for indexing
      ...Indexing
      ✓ Indexed 647 packages
      ✓ Provenance obtained from attestation
  Error: API operation failed: Message: 403 Forbidden; body: "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>403 Forbidden</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Forbidden</h1>\n<h2>Your client does not have permission to get URL <code>/v1/graphql</code> from this server.</h2>\n<h2></h2>\n</body></html>\n", Locations: [], Extensions: map[code:request_error], Path: []

Please let me know if you need additional information

mikeparker commented 2 months ago

👋 Hi Pavel! Hope you're doing well!

This looks similar to https://github.com/docker/scout-cli/issues/89#issuecomment-1932367067. Our internal conversations say this isn't anything we recognise our side, maybe a proxy or something similar?

We did a bit of a github search for the error message, which didn't throw up much but we did see this unless its related to your registry https://github.com/kubernetes/registry.k8s.io/blob/218104931605223efecdbbf36f338cac3f6b3b12/docs/debugging.md?plain=1#L10

I'm pretty sure the v1/graphql is a call to the Scout API hosted on GCP and there are reports online of Google giving that 403 error for various reasons like a blocked IP address, calls from a sanctioned country (e.g. Iran), so perhaps there is something going on there. Does it always fail for the same image e.g. if you retry the action? Or is it sporadic even for the exact same call? I'm wondering if there is some kind of github <-> google rate limiting going on. My guess is that you've successfully pulled the image, extracted the SBOM then the Scout CLI connects to our API to match the SBOM packages to the latest CVE information and thats the call thats failing.

You said you're self hosting the runners. How many runners do you have, is it possible there is one of them with a dodgy IP address that's being mistaken for somewhere sanctioned?

velll commented 2 months ago

Thank you for the quick reply Mike! ⚡

We do self host runners for these workflows in our Azure cloud in EU 🇪🇺 I will try reproducing this directly from the runners next week, and see if changing the IPs helps.

The action does actually consistently fail for the same image, so I don't immediately suspect network. But there very much could be something interesting there.

Thank you for all the details! I will continue next week and will post the findings

velll commented 2 months ago

Took us a while to get to it, but we basically confirmed that this error is related to a banned IP.

We build our images on self hosted runners, and for one of the workflows we had only one machine operating runners with the specific set of labels. This machine was unlucky to get an IP that was banned, that's why we were seeing the errors consistently.

We confirmed this by running docker scout commands directly from the machines hosting the runners (docker/scout-cli image was extremely useful for running commands on a server while having a minimal footprint).

the old machine — consistent 403s
a fresh machine configured exactly the same — no errors

We're confident, that the only difference in the configuration is the IP address. We replaced the old machine, and didn't encounter these errors since.

I'm closing the issue. Thank you for the support! Hope this thread helps the next team running into a similar issue.

docker / scout-action