failed to fetch artifacts: failed to list artifacts of repository

[AyushCloud]

Unable to replicate the repository which has count of 152+ repos. Harbor VM 2 Scalable VM 2 Database(Postgres) VM 1 Redis VM 1 Harbor Version 2.0.0 Docker version 19.0.0 docker-compose version 1.26

Error while replication in core.log : 2020-10-16T13:19:13Z [ERROR] [/replication/operation/controller.go:103]: the execution 14 failed: failed to fetch artifacts: failed to list artifacts of repository 'ayush/repo/image-name': http error: code 403, message {"errors":[{"code":"FORBIDDEN","message":"forbidden"}]}

Please let me know if you any more information

Thank You

[marthydavid]

+1

Env: 3 node "VM" Postgres Master-2Slave setup. Minio for S3 storage docker-compose version: 1.24.0 Harbor Version: 2.1.0 Docker Version: 19.03.8 Try to replicate between two harbor "cluster" instance: core.log: Oct 19 06:08:54 192.168.16.1 core[25360]: 2020-10-19T04:08:54Z [ERROR] [/replication/operation/controller.go:103]: the execution 51398 failed: failed to fetch artifacts: failed to list artifacts of repository 'hosted/k8s-netchecker-server': http error: code 504, message #015

[marthydavid]

From my point of view the biggest problem is occuring when the docker image name is like that: projectname/image-name-part1/image-name-part2

The parser is using %252Fmaybe that's the problem i'm just try to figure it out.

I've try to replicate a repo with images that has only 1 namepart.

[AyushCloud]

Could be possible. For me with one image-part is working fine.

[chlins]

@AyushCloud Hi, Can you provide more info such as registry configuration and replication rule details.

[AyushCloud]

Hello @chlins,

I am not sure what exactly you meant by the registry configuration. Could you please specify the exact configuration you want to know. PFA the snippet of replication rule.

[chlins]

@AyushCloud Thanks，registry configuration includes adapter type, and if use credentials etc... which can be found below.

[AyushCloud]

Hello @chlins,

I have attached the information as mentioned.

The issue is that destination repo has images in below format: 1) projectname/image-name-part1/image-name-part2 2) projectname/image-name-part1

So when I pull the images seperately, then the replication is working fine but with a single rule it is giving the error. 1) projectname/*  working, able to pull only the images with projectname/image-name-part1 (69 repos) 2) projectname/image-name-part1/  working, able to pull only the images with projectname/image-name-part1/image-name-part2(82 repos) 3) projectname/  not working

I am not sure if it is related to the postgresql max connection which is 100 by default.

[chlins]

@AyushCloud Which your source registry access id? is it admin account or other account?

[AyushCloud]

Hello @chlins

I am using the admin account for all the replication requests

[chlins]

@AyushCloud I have tested the case like your replication rule it worked well, but my harbor does not have so much images, but i think the logic is same whatever how many images, can you try to create a new project, and push several images which contains projectname/image1 and projectname/path1/image2, and test your replication rule again in this new project?

[tangkelu]

1.From Harbor domainA/lib/image-name:tag to Harbor domainB/lib/image-name:tag, failed 403 FORBIDDEN 2.change the replication rule ,From Harbor domainA/lib/image-name:tag to Harbor domainB/lib-sync/mage-name:tag, Success. like a bug

[chlins]

@ywk253100 Can you help to check this issue, It looks a little strange when harbor replicate images from harbor adapter.

[AyushCloud]

I am trying to replication pull from a source repository having 110+images(300GB), but getting the error: the execution 1 failed: failed to fetch artifacts: failed to list artifacts of repository 'ayush/': http error: code 500, message {"errors":[{"code":"UNKNOWN","message":"internal server error"}]}

[AyushCloud]

On checking the core.log on the source registry i found the connection error: r.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /api/v2.0/projects/ayush/repositories/image-name/artifacts request in transaction failed: pq: sorry, too many clients already"}]} [ERROR] [/core/config/config.go:131]: failed to load config, error pq: remaining connection slots are reserved for non-replication superuser connections [WARNING] [/server/middleware/security/security.go:55][requestID="0a0ca926-2145-483d-94f2-6d23c716fde5"]: failed to get auth mode: pq: remaining connection slots are reserved for non-replication superuser connections [ERROR] [/lib/orm/orm.go:69]: begin transaction failed: pq: sorry, too many clients already [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /api/v2.0/projects/ayush/repositories/image-name/artifacts request in transaction failed: pq: sorry, too many clients already"}]} [ERROR] [/core/config/config.go:131]: failed to load config, error pq: remaining connection slots are reserved for non-replication superuser connections

[chlins]

On checking the core.log on the source registry i found the connection error: r.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /api/v2.0/projects/ayush/repositories/image-name/artifacts request in transaction failed: pq: sorry, too many clients already"}]} [ERROR] [/core/config/config.go:131]: failed to load config, error pq: remaining connection slots are reserved for non-replication superuser connections [WARNING] [/server/middleware/security/security.go:55][requestID="0a0ca926-2145-483d-94f2-6d23c716fde5"]: failed to get auth mode: pq: remaining connection slots are reserved for non-replication superuser connections [ERROR] [/lib/orm/orm.go:69]: begin transaction failed: pq: sorry, too many clients already [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /api/v2.0/projects/ayush/repositories/image-name/artifacts request in transaction failed: pq: sorry, too many clients already"}]} [ERROR] [/core/config/config.go:131]: failed to load config, error pq: remaining connection slots are reserved for non-replication superuser connections

@AyushCloud From your logs, maybe the database connections of your harbor instance was fulled, you can check the FAQs https://github.com/goharbor/harbor/wiki/Harbor-FAQs#modify-the-max-connections-of-the-database-for-versions-less-than-v202 and try if it can solve your problem.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Joseph94m]

I'm having the same problem, even though I set max_open_conn in harbor to 50 and max_idle_conn to 10, and my PGSQL max connection setting is set to 100.

I still get the:

2024-07-29T07:06:26Z [ERROR] [/controller/artifact/controller.go:740]: failed to list accessories of artifact 93523: failed to connect to `host=XXXXX user=XXX database=XXX`: server error (FATAL: remaining connection slots are reserved for non-replication superuser connections (SQLSTATE 53300))
2024-07-29T07:06:26Z [WARNING] [/common/rbac/project/evaluator.go:80]: Failed to get info of project 5 for permission evaluator, error: failed to connect to `host=XXXXXl user=XXX database=XXX`: server error (FATAL: remaining connection slots are reserved for non-replication superuser connections (SQLSTATE 53300))

I suspect that Harbor may not be respecting its configuration