Closed spielkind closed 2 years ago
jvanz
commented
2 years ago I'm doing some tests and the issue seems to be with the oci-distribution. Even the example program from the library fail to pull the manifest from the repo. You can see the error with the following command:
cargo run --no-default-features --features "rustls-tls" --example get-manifest --verbose mtr.devops.telekom.de/reik_keutterling/mytest:signedI'll continue the investigation and fix the issue.
spielkind
commented
2 years ago Thx, for looking into it. I just tried it with cosign, also kyverno had no issues with it. But it's probably because of how quay was configured / setup, just let me know if this a "issue on the server side". Afaik the colleagues run some kind of WAF in front of the registry, which may also can cause some issues.
As you mentioned I got the same issue when running the example get-manifest from the library.
❯ cosign triangulate mtr.devops.telekom.de/reik_keutterling/mytest:signed
mtr.devops.telekom.de/reik_keutterling/mytest:sha256-4998620d0c06810f7e021e789ebbdc145bf6e5c774ddd4f4c05f64f85bd40fc2.sig
❯ cosign verify --key cosign.pub mtr.devops.telekom.de/reik_keutterling/mytest:signed
Verification for mtr.devops.telekom.de/reik_keutterling/mytest:signed --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
[{"critical":{"identity":{"docker-reference":"mtr.devops.telekom.de/reik_keutterling/mytest"},"image":{"docker-manifest-digest":"sha256:4998620d0c06810f7e021e789ebbdc145bf6e5c774ddd4f4c05f64f85bd40fc2"},"type":"cosign container image signature"},"optional":null}]Okay, one more update. The oci-distribution is failing because it trys to authenticate before requesting the manifest. So, when it trys to access https://mtr.devops.telekom.de/v2/auth the response is http status code 403.
Failed to authenticate for image 'Reference { registry: "mtr.devops.telekom.de", repository: "reik_keutterling/mytest", tag: Some("signed"), digest: None }': <html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>If I remove the code calling the authentication method, I can fetch the manifest:
OCI Image Manifest( schema-version: '2', media-type: 'application/vnd.docker.distribution.manifest.v2+json', config: '( media-type: 'application/vnd.docker.container.image.v1+json', digest: 'sha256:2724067da075182f23aa386e0f62aeada1e0abf0a04e1cc421ff5f4086bf3e1e', size: '1632', urls: '[]', annotations: '{}' )', layers: '["( media-type: 'application/vnd.docker.image.rootfs.diff.tar.gzip', digest: 'sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8', size: '2798889', urls: '[]', annotations: '{}' )"]', annotations: '{}' )Can you check if the FAW is blocking the auth endpoint?
Furthermore, about your question:
Bonus question: Is it possible to verify private images which require specific pull secerts? Couldn't find any informations on that.
You can define the pull secrets in the policy server definition
spielkind
commented
2 years ago I asked the colleagues about the configuration for the WAF, they told me that they currently work on it. But I also think its related to the WAF. When I curled the URL through my local workstation it worked for me. While using an online tool like https://reqbin.com/ I also get the 403 forbidden.
curl -i https://mtr.devops.telekom.de/v2/auth
HTTP/2 401
date: Mon, 20 Jun 2022 14:54:15 GMT
content-type: application/json
content-length: 112
set-cookie: quay.session=599d2b1b0cd63c723045fe08892a60d9|c791a42e6919ba98c33df5b8367872d1; Path=/; Secure; HttpOnly
www-authenticate: Bearer realm="https://mtr.devops.telekom.de/v2/auth",service="mtr.devops.telekom.de"
docker-distribution-api-version: registry/2.0
strict-transport-security: max-age=15724800; includeSubDomains
{"errors":[{"code":"UNAUTHORIZED","detail":{},"message":"access to the requested resource is not authorized"}]}Regarding private images, thanks for feedback is there a solution for having multiple pull secrets handled by different persons, distributed through the whole cluster? - See also Slack for details.
jvanz
commented
2 years ago I asked the colleagues about the configuration for the WAF, they told me that they currently work on it. But I also think its related to the WAF. When I curled the URL through my local workstation it worked for me. While using an online tool like https://reqbin.com/ I also get the 403 forbidden.
@flavio what do you think on changing the oci-distribution adding a configuration/flag to skip the auth even if the request is not authenticated? I understand that we firewall should be fixed. But I'm wondering if make sense to force oci-distribution authenticate every time it is not authenticated. Is this a OCI requirement in the OCI spec? (I'll try to figure out that as well)
Regarding private images, thanks for feedback is there a solution for having multiple pull secrets handled by different persons, distributed through the whole cluster? - See also Slack for details.
For future readers, let me document here what I just sent to you in the Slack channel:
unfortunately, there is no other way to define the pull secrets. You can deploy multiple policy servers with different pull secret. I understand that is not the best solution. But it is a possible solution for now. Let me take this opportunity to ask you, what do you like to have? Secrets defined in cluster? Thus, multiple policy servers can refer to the same secret? Let us know what do you think. I cannot guarantee that we can do that now, but your feedback, as user, is very important.
spielkind
commented
2 years ago @jvanz the colleagues applied yesterday a change at the WAF, while cargo still fails with "403 forbidden" for me curl (as before) and https://reqbin.com/ now work for https://mtr.devops.telekom.de/v2/auth
Edit: Seems they, changed sth. else, as the cargo get-manifest seems to work now for me.
jvanz
commented
2 years ago @jvanz the colleagues applied yesterday a change at the WAF, while cargo still fails with "403 forbidden" for me curl (as before) and https://reqbin.com/ now work for https://mtr.devops.telekom.de/v2/auth
Edit: Seems they, changed sth. else, as the cargo get-manifest seems to work now for me.
Nice! I can also see here that the example program is working. Is the policy server working as well?
spielkind
commented
2 years ago Yes, also the policy server is working now.
Thanks for your support!
Is there an existing issue for this?
Current Behavior
Signature couldn't be verified.
Expected Behavior
Verify signature. :)
Steps To Reproduce
Test policy:
Test Image:
mtr.devops.telekom.de/reik_keutterling/mytest:signedEnvironment
Anything else?
Debug output:
Seems like he can't fetch the layer:
2022-06-15T12:11:06.926783Z DEBUG reqwest::async_impl::client: response '403 Forbidden' for https://mtr.devops.telekom.de/v2/reik_keutterling/mytest/blobs/sha256:2f51aad9ad42e0788893eabe3422d13e97a2e656c17da1ef7eb5cc9b2ca154f6But as this a public image this should work:
I guess it's related to the redirect & session cookie.
Bonus question: Is it possible to verify private images which require specific pull secerts? Couldn't find any informations on that.