aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.56k stars 374 forks source link

Publish Switch to S3 always lists all objects in bucket #1181

Closed Htgan closed 8 months ago

Htgan commented 1 year ago

aptly publish switch to S3 takes a significant amount of time due to aptly pulling a list of every object in s3 most are not from the repository we're updating

S3PublishEndpoints configured without prefix so this is the root of the bucket. I have a large number of repositories inside 1 bucket, distinguished like so: focal-updates/ jammy-cherry/ xenial/ stable/ unstable/

download_prefix=stable Upon aptly publish switch -skip-contents -skip-cleanup $distro_name s3:$endpoint:$download_prefix $snapshot_name, aptly will list objects in the entire bucket instead of only what's in stable/

No issues with the repositories. It just takes a long time to switch a snapshot (and longer as the bucket grows in size). Wondering if I've misunderstood how I'm supposed to set up the repositories in S3.

Any guidance will be greatly appreciated.

Htgan commented 1 year ago

I've been getting around this issue by creating additional s3 endpoint entries in aptly.conf It appears upon publish switch, the prefix specified in the command: aptly publish switch <distribution> [[<endpoint:>]<prefix>] <new-snapshot> is ignored when s3/ListObjects is called. If no prefix is set in the conf, then this request will always list all objects in root of S3. What I've had to do was duplicate the s3 endpoint section for each prefix we have, specify the prefix in the s3 endpoint, and do not specify the prefix in the aptly publish switch command. This is quite tedious but gets the job done for now.