Ceph support

[rofafor]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior: Garbage collection fails when using Ceph as S3 storage:

Jun 14 17:59:22 172.27.0.1 registryctl[26877]: 2019-06-14T14:59:22Z [ERROR] [/registryctl/api/registry.go:49]: Fail to execute GC: exit status 2, command err: panic: runtime error: invalid memory address or nil pointer dereference
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc51d8d]
Jun 14 17:59:22 172.27.0.1 registryctl[26877]:
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: goroutine 1 [running]:
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk.func1(0xc000404f80, 0xc000368c01, 0xc0004e76e8)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:973 +0x9d
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc00014a100, 0x7fb825965100, 0xc00019f650, 0xc00017c280, 0xc0002657e8, 0x0, 0x0, 0x0, 0x1, 0x2)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:4198 +0x111
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk(0xc000404f00, 0xff4d80, 0xc00019f5e0, 0xc0004e78a8, 0xc0003b41b1, 0x20, 0xea98b5, 0x1, 0xc00028f380, 0x0, ...)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:971 +0x3a1
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).Walk(0xc000404f00, 0xff4d80, 0xc00019f5e0, 0xc0001420c0, 0x20, 0xc00028f380, 0x2, 0x0)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:919 +0x160
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry/storage/driver/base.(*Base).Walk(0xc00033ce00, 0xff4d80, 0xc00019f5e0, 0xc0001420c0, 0x20, 0xc00028f380, 0x0, 0x0)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go:239 +0x234
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry/storage.(*registry).Enumerate(0xc00019f570, 0xff4ac0, 0xc0003f1ad0, 0xc000368bc0, 0x0, 0x0)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/storage/catalog.go:67 +0xe3
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry/storage.MarkAndSweep(0xff4ac0, 0xc0003f1ad0, 0x1000c20, 0xc00033ce00, 0xff6c80, 0xc00019f570, 0x100, 0xff6c80, 0xc00019f570)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/storage/garbagecollect.go:40 +0x1e7
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/registry.glob..func3(0x171eca0, 0xc00042f820, 0x1, 0x2)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/registry/root.go:80 +0x48a
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x171eca0, 0xc00042f740, 0x2, 0x2, 0x171eca0, 0xc00042f740)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x191
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x171efe0, 0xc000233f88, 0xc00008c058)
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0x2f4
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: main.main()
Jun 14 17:59:22 172.27.0.1 registryctl[26877]: #011/go/src/github.com/docker/distribution/cmd/registry/main.go:23 +0x2d

This is due to a bug in Docker registry that's been finally fixed in the master branch: https://github.com/docker/distribution/pull/2879

I validated the fix by cloning the current docker/distribution master (79f6bcb), building the registry binary and replaced the one in the goharbor/harbor-registryctl-v1.8.1 image and finally the garbage collection is working fine.

I recommend that this small fix will be cherry-picked into Harbor's next release unless there aren't any newer registry releases than the current 2.7.1.

Steps to reproduce the problem:

1. Configure Ceph as S3 storage
2. Execute Garbage Collection via UI or via command-line inside registryctl container.

Versions: Please specify the versions of following systems.

• harbor version: [1.8.1]
• docker engine version: [18.09.6]
• docker-compose version: [1.24.0-rc3]

[bitsf]

This is a good suggestion, thanks. Currently we are using docker registry distribution directly, not build locally yet. But maybe we'll have to do this in future if we have much more important patches that Harbor extremely need and we can't wait too long.

[rofafor]

IMO, the faulty S3 interface is used only during garbage collection in registryctl and the required binary is built from the docker registry distribution version control. Adding support for patches into this phase would be quite clean implementation: https://github.com/goharbor/harbor/blob/master/make/photon/registry/builder#L23 https://github.com/goharbor/harbor/blob/master/make/photon/registryctl/Dockerfile#L12

[bitsf]

@rofafor Yes I know, but currently we didn't have a place to maintain the patch.

@reasonerjt has suggested to fork the docker & registry git repo, and maintain our own branch, but this still need some effort to think about how to organize the build process.

[reasonerjt]

@xaleeks since it's merged to master of upstream registry, we may consider to cherrypick and compile it to include the fix?

[bitsf]

Shouldn't we directly use the newest upstream registry, not just cherrypick

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.