search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
23.61k stars 4.72k forks source link

Harbor Replication fails with PROJECT POLICY VIOLATION though image is signed #20768

Open MinerYang opened 1 month ago

MinerYang commented 1 month ago

Discussed in https://github.com/goharbor/harbor/discussions/20752

Originally posted by **TnGadaria** July 17, 2024 Hi I have a registry with two replicas and am using notation to sign the image. I am facing an issue with the `replication` towards both of the registries, as per the replication logs it does not find that the image is signed and fails with `PROJECT POLICY VIOLATION`, however, I am pushing the image and then signing the same, and expecting the image to replicate after signing. I have noticed many times that, after pushing and signing the image to the registry, the replication registries create the repository if it's not there but does not replicate any artifacts. This is an intermediate issue, however, observed frequently. Any help would be appreciated. Harbor version: 2.11.0 Notary version: 1.1.0 Logs for more info: ``` 2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created 2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: https://165.225.18.211, insecure: true] created 2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying opsarch-private/alpine-fix:[v1,sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659,sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32](source registry) to opsarch-private/alpine-fix:[v1,sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659,sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32](destination registry)... 2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying opsarch-private/alpine-fix:v1(source registry) to opsarch-private/alpine-fix:v1(destination registry)... 2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact opsarch-private/alpine-fix:v1 ... 2024-07-17T11:50:24Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact opsarch-private/alpine-fix:v1 pulled 2024-07-17T11:50:25Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:2b76c9374f7571d8faca1b7cbe435aa8c9ebfc3a0d94c15f36f188a0cd15e2b9(the 1th running)... 2024-07-17T11:50:25Z [DEBUG] [/controller/replication/transfer/image/transfer.go:392]: the blob size is 1467 bytes 2024-07-17T11:50:25Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:2b76c9374f7571d8faca1b7cbe435aa8c9ebfc3a0d94c15f36f188a0cd15e2b9 completed 2024-07-17T11:50:25Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99(the 1th running)... 2024-07-17T11:50:25Z [DEBUG] [/controller/replication/transfer/image/transfer.go:392]: the blob size is 3623844 bytes 2024-07-17T11:50:26Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99 completed 2024-07-17T11:50:26Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:4e997a7a8a24a8d55e3d7b3816f1fdf016afa475534988e70bc65ab63bf1e61d(the 1th running)... 2024-07-17T11:50:26Z [DEBUG] [/controller/replication/transfer/image/transfer.go:392]: the blob size is 4607237 bytes 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:4e997a7a8a24a8d55e3d7b3816f1fdf016afa475534988e70bc65ab63bf1e61d completed 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact opsarch-private/alpine-fix:v1 ... 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact opsarch-private/alpine-fix:v1 pushed 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy opsarch-private/alpine-fix:v1(source registry) to opsarch-private/alpine-fix:v1(destination registry) completed 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659(source registry) to opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659(destination registry)... 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659 ... 2024-07-17T11:50:27Z [ERROR] [/controller/replication/transfer/image/transfer.go:470]: failed to pull the manifest of artifact opsarch-private/alpine-fix:sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659: http status code: 412, body: {"errors":[{"code":"PROJECTPOLICYVIOLATION","message":"The image is not signed by notation."}]} 2024-07-17T11:50:27Z [ERROR] [/controller/replication/transfer/image/transfer.go:194]: http status code: 412, body: {"errors":[{"code":"PROJECTPOLICYVIOLATION","message":"The image is not signed by notation."}]} 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32(source registry) to opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32(destination registry)... 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32 ... 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32 pulled 2024-07-17T11:50:27Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact opsarch-private/alpine-fix:sha256:91941698761dbed7a92add7e62c6ef2832c52c9e1a58460de60b145622758b32 already exists on the destination registry, skip 2024-07-17T11:50:27Z [ERROR] [/controller/replication/transfer/image/transfer.go:200]: got error during the whole transfer period, mark the job failure ```
MinerYang commented 1 month ago

Hi @TnGadaria,

Please provide bellowing output

  1. curl v2/referrer api 
    curl -k -u 'admin:password' https://<harbor-hostname>/v2/opsarch-private/alpine-fix/sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659
  2. check the existence of signature and kindly provide the output 
    docker exec -it <db-pod> /bin/bash
psql -d registry
select * from artifact_accessory where subject_artifact_digest='sha256:6ec6600f1d51b51480ddfcc23194c007da7e02eb2162388e7affc607e97f6659'
  3. obtain the above digest as signature digest and check it's manifest, take sha256:123xxxx 
    docker exec -it <registry-pod> /bin/bash
/storage/docker/registry/v2/blobs/sha256
cat 12/123xxx/data