search

elastisys / compliantkubernetes-apps

Elastisys Compliant Kubernetes is an open source, Certified Kubernetes distribution designed according to the ISO27001 controls: providing you with security tooling and observability from day one.
https://elastisys.io/compliantkubernetes/
Apache License 2.0
45 stars 7 forks source link

Check if the S3 API can be used for Harbor on Cleura #481

Open simonklb opened 3 years ago

simonklb commented 3 years ago

What should be investigated. We currently use Swift as the Harbor image/chart storage because there were some issues when uploading/downloading large images using the S3 API. This was introduced over a year ago and things may have been fixed in the meantime.

Let's investigate if S3 can be used as this would remove some extra configuration and management burden.

What artifacts should this produce. Try uploading charts and images (especially really large ones) to Harbor while using S3 on CityCloud. If this works, create a follow up feature issue.

aarnq commented 1 year ago

Have tested on Cleura / Fra1 and it worked as expected so we should be able to start using S3 as default on Cleura.

OlleLarsson commented 1 year ago

Update: do remember to check if the object naming convention (?) discrepancy between swift and s3 is a problem.

viktor-f commented 1 year ago

If we can switch to s3, then we also need see if we need some migration path for environments that currently use swift.

viktor-f commented 1 year ago

I have tried installing harbor on cleura using s3 and can confirm that it works to push large images (tested with 20GB image that has one layer larger than 7GB).

For the migration there is at least one significant difference in how harbor uses the object storage. When using swift it places some files at /files/docker/ while when using s3 it places them at /docker/. It is possible to simply move the files to the correct "folder" and then it seems to work, I tried pulling an image and scanning an image that was uploaded before the migration.

To move the files: s3cmd --config <(sops -d .state/s3cfg.ini ) cp s3://<bucket-name>/files/docker/ s3://<bucket-name>/docker/ -r with the correct <bucket-name>

However, we would still have the issue that rclone cannot sync all files using s3 that were created by harbor using swift. So in order to make that work, we might have to rclone the harbor bucket using swift to a new harbor bucket using s3. Then we should have files that we can then rclone with s3.

Since harbor registry will drop support for swift, I have opened an issue in the harbor git repo to see if they know of any other potential issues and if they are planning on providing a migration path for this. I will keep this task pending for a while and see if we get any response there, I assume that it will take a while before the support is dropped so this should not be very urgent.